Facilitating Interaction between Users and their Environments Using a Headset having Input Mechanisms

ABSTRACT

A headset is described herein for presenting audio information to the user as the user interacts with a space, e.g., as when a user navigates over a route within a space. The headset may include a set of input mechanisms for receiving commands from the user. The commands, in turn, invoke respective space-interaction-related functions to be performed by a space interaction (SI) module. The headset may operate with or without a separate user computing device.

This application claims the benefit of U.S. Provisional Application No.62/073,762 (the '762 application), filed Oct. 31, 2014. The '762application is incorporated by reference herein in its entirety

BACKGROUND

A user may rely on various conventional mechanisms in generating andexecuting travel plans and/or in exploring his or her environment in amore spontaneous and unconstrained manner. For example, a user may use aroute selection application to generate a route for use in travelingbetween two locations. That same application may then guide the user asthe user travels over the route through a series of prompts. The promptsmay correspond to spoken and/or displayed messages, such as a messagethat verbally instructs the user to make a turn within a prescribeddistance. There is nevertheless considerable room for improvement inthese conventional mechanisms.

SUMMARY

Various tools are described herein for assisting a user in interactingwith physical and/or virtual spaces. Such assistance, for example, mayfacilitate a user's navigation within the physical and/or virtualspaces. As used herein, navigation refers a user's purposeful movementthrough a space according to a predetermined plan, and/or movement thatmay reflect spontaneous decisions that do not necessarily conform to anypredetermined plan. The tools may also assist a user's explorationwithin a space at any given moment in time, e.g., corresponding to aprocess by which the user orients himself or herself in the space.

As a general principle, the tools are designed to provide highlyinformative and timely assistance to the user in a user-friendly manner,thereby enriching the user's experience of the physical and virtualspaces through which the user moves, but without unduly distracting theuser from the actual task of interacting with the spaces. In view ofthese characteristics, the tools may be successfully used even by userswith vision impairments and/or other handicaps. However, the tools aregeneral-purpose in nature, and thus may provide user-friendly andunobtrusive guidance to any user in performing any task in which theuser interacts within his or her environment.

The tools have various aspects which contribute to the above-summarizedperformance. According to one aspect, a headset is described herein forpresenting audio information to the user as the user interacts with thespace. The headset may also include a set of input mechanisms forreceiving commands from the user. The commands, in turn, invokerespective space-interaction-related functions to be performed by aspace interaction (SI) module.

For instance, commands may instruct the SI module to: stop deliveringthe audio information; repeat a last-delivered item of audioinformation; repeat a set of last-delivered items of audio information(e.g., by “rewinding”); start an explore mode; start an orientationmode; perform a more-information function (to request additionalinformation); start or stop a listening mode (in which voice recognitionis enabled and disabled, respectively), and so on. In the explore mode,the space interaction module provides information regarding a set ofitems of interest that are associated with a subspace to which attentionof the user is currently directed. In the orientation mode, the spaceinteraction module provides information regarding a set of items ofinterest that are associated with an entire space round the user, at acurrent time.

In one implementation, the headset may include its own orientation,motion, and/or position determination mechanism(s) (e.g., a compass). Inanother implementation, the headset relies on a separate orientation,motion, and/or position determination mechanism(s) provided by aseparate user device (e.g., a smartphone).

In one implementation, a system implements the SI module usingprocessing resources associated with the headset alone. In anotherimplementation, the system implements the SI module using processingresources associated with the separate user computing device alone. Inother implementations, the functions performed by the SI module may beshared by any of the headset, and/or the user computing device, and/orone or more remote processing resources (e.g., “cloud-implemented”services).

The above features contribute to the above goal of allowing the user tosafely and efficiently move through his or her environment. Forinstance, the features provide a convenient way by which the user mayactivate various operational modes (e.g., by interacting with the userinput mechanisms of the headset) without unduly distracting the user asthe user moves through the environment, e.g., without requiring the userto access and interact with a separate user device.

The above approach can be manifested in various types of systems,devices, components, methods, computer readable storage media, datastructures, graphical user interface presentations, articles ofmanufacture, and so on.

This Summary is provided to introduce a selection of concepts in asimplified form; these concepts are further described below in theDetailed Description. This Summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an overview of a system for assisting a user in interactingwith a physical and/or virtual space.

FIG. 2 shows an example in which the user uses the system of FIG. 1 totake a journey that is defined by plural waypoints. In other cases, theuser may use the system to interact with an environment in a moreopen-ended and spontaneous manner.

FIG. 3 shows an overview of computing functionality that may be used toimplement the system of FIG. 1.

FIG. 4 shows one implementation of a space interaction module, which isa component of the computing functionality of FIG. 3.

FIG. 5 shows one implementation of an application interface module,which is a component of the space interaction module of FIG. 4.

FIG. 6 shows an example of a headset, handheld user computing device(“user device” for brevity), and another user computing device; thesedevices may implement one or more aspects of the system of FIG. 1.

FIG. 7 shows one manner of implementing the headset and user device ofFIG. 6.

FIG. 8 shows another manner of implementing the headset and the(optional) user device of FIG. 6.

FIG. 9 shows a scenario in which a user interacts with the user deviceof FIG. 6 within a vehicle.

FIG. 10 is a flowchart which illustrates one manner of operation of theequipment shown in FIG. 6.

FIG. 11 shows different workspaces that may be presented by theapplication interface module of FIGS. 4 and 5, e.g., in a user interfacepresentation provided by a display output mechanism of a user device.

FIG. 12 shows a gesture that may be used to transition from a firstworkspace to a second workspace.

FIG. 13 shows a default hub menu that may be presented in a homeworkspace, according to one implementation.

FIGS. 14-16 show respective context menus that may be presented in thehome workspace.

FIG. 17 shows a menu that identifies a plurality of functions that maybe invoked; that menu may be presented in a main workspace.

FIG. 18 shows a menu that a user may use to set various parameters; thatmenu may be presented in a settings workspace.

FIG. 19 shows a menu that a user may use to access information that isrelevant to the user as he or she conducts a journey or otherwiseinteracts with an environment; that menu may be presented in aninformation workspace.

FIG. 20 shows a menu that a user may use to access informationpertaining to nearby items of interest (IOIs); that menu may bepresented in a “nearby me” workspace.

FIG. 21 shows an example of a transient menu that may be presented inthe home workspace.

FIG. 22 shows an example of an overlay menu that, in this particularcase, is presented in the settings workspace.

FIG. 23 demonstrates gestures that a user may use to obtain audioinformation regarding his or her current context.

FIG. 24 shows a gesture that a user may use to activate any menu in anyworkspace.

FIG. 25 shows a gesture that a user may use to navigate through acollection of menu items in any menu.

FIG. 26 shows a back gesture that a user may use to navigate from acurrent menu to some other preceding menu.

FIG. 27 shows an alternative organization of workspaces, compared to theimplementation of FIG. 11.

FIG. 28 shows tab information in a home workspace of FIG. 27,corresponding to a state in which a user has not yet created any tabs.

FIG. 29 shows a tab menu in the home workspace of FIG. 27, correspondingto a state in which a user has now created a collection of tabs.

FIG. 30 shows an alternative manner of organizing menu items within amenu, compared to the implementation of FIG. 25.

FIGS. 31 and 32 show an alternative manner of scrolling through menuitems within a menu, and then selecting a menu item, compared to theimplementation of FIG. 25.

FIG. 33 shows a back gesture performed at the periphery of a menu, andan action that is performed in response to this gesture.

FIG. 34 shows a gesture that is performed by drawing a circle shape on amenu, and a back action that is performed in response to the gesture.

FIG. 35 shows a gesture that is performed by drawing a half-circle on amenu, and a back action that is performed in response to the gesture.

FIGS. 36 and 37 show different gestures that can be used to increase anddecrease, respectively, a level of verbosity provided by the system ofFIG. 1, and a level of contextual information provided by the system.

FIG. 38 is a flowchart which summarizes one manner of operation of theapplication interface module of FIGS. 4 and 5.

FIG. 39 demonstrates the use of three-dimensional audio information tocreate a perception of sound which emanates from a particular locationwithin space.

FIG. 40 demonstrates the use of three-dimensional audio information tocreate a perception of sound that moves across a series of locationswithin space.

FIG. 41 demonstrates a manner in which a path guidance module can usethree-dimensional sounds (e.g., a periodic beat sound) to guide the userin a desired direction.

FIG. 42 demonstrates a manner in which an exploration module can usethree-dimensional sounds to identify locations of IOIs that areassociated with a current focus of interest of the user.

FIG. 43 demonstrates a manner in which an orientation module can usethree-dimensional sounds to identify IOIs that are associated with anentire space around the user.

FIG. 44 is a flowchart that describes the use of three-dimensionalsounds to assist the user in interacting with a space.

FIG. 45 is a flowchart that describes one manner in which the pathguidance module (of FIGS. 4 and 41) may use the three-dimensional soundsto guide the user along a desired route.

FIG. 46 is a flowchart that describes one manner in which the spaceinteraction module can use three-dimensional sounds to identify IOIs,e.g., in an automated mode (e.g., while the user traverses a route orotherwise moves in space), in an explore mode, or in an orientationmode.

FIG. 47 shows an environment having a plurality of beacons. In oneillustrative implementation, the beacons produce non-overlapping rangesin which their respective signals may be detected by a user who movesacross the environment or otherwise interacts with the environment.

FIGS. 48-50 show other environments, each having a greater number ofbeacons compared to the example of FIG. 47.

FIG. 51 is a flowchart that shows one manner of operation of abeacon-based guidance module, within the context of the types ofenvironments of FIGS. 47-50.

FIG. 52 is a flowchart that provides further details regarding onemanner in which the beacon-based guidance module can determine thecurrent location of the user within an environment.

FIG. 53 shows illustrative computing functionality that can be used toimplement any aspect of the features shown in the foregoing drawings.

The same numbers are used throughout the disclosure and figures toreference like components and features. Series 100 numbers refer tofeatures originally found in FIG. 1, series 200 numbers refer tofeatures originally found in FIG. 2, series 300 numbers refer tofeatures originally found in FIG. 3, and so on.

DETAILED DESCRIPTION

This disclosure is organized as follows. Section A provides an overviewof a system for assisting a user in interacting with a real and/orvirtual space. Section B describes different types of headset andhandheld user computing devices that can be used in the system ofSection A. Section C describes an illustrative user interface experiencethat may be provided by the system of Section A. Section D describesfunctionality for generating and using three-dimensional sounds, andother (non-three-dimensional) types of sounds. Section E describesbeacon-based guidance functionality for assisting the user in navigatingthrough an environment that is populated with beacons having, in oneillustrative implementation, non-overlapping ranges. And Section Fdescribes illustrative computing functionality that can be used toimplement any aspect of the features described in the previous sections.

As a preliminary matter, some of the figures describe concepts in thecontext of one or more structural components, variously referred to asfunctionality, modules, features, elements, etc. The various componentsshown in the figures can be implemented in any manner by any physicaland tangible mechanisms, for instance, by software running on computerequipment, hardware (e.g., chip-implemented logic functionality), etc.,and/or any combination thereof. In one case, the illustrated separationof various components in the figures into distinct units may reflect theuse of corresponding distinct physical and tangible components in anactual implementation. Alternatively, or in addition, any singlecomponent illustrated in the figures may be implemented by plural actualphysical components. Alternatively, or in addition, the depiction of anytwo or more separate components in the figures may reflect differentfunctions performed by a single actual physical component. FIG. 53, tobe described in turn, provides additional details regarding oneillustrative physical implementation of the functions shown in thefigures.

Other figures describe the concepts in flowchart form. In this form,certain operations are described as constituting distinct blocksperformed in a certain order. Such implementations are illustrative andnon-limiting. Certain blocks described herein can be grouped togetherand performed in a single operation, certain blocks can be broken apartinto plural component blocks, and certain blocks can be performed in anorder that differs from that which is illustrated herein (including aparallel manner of performing the blocks). The blocks shown in theflowcharts can be implemented in any manner by any physical and tangiblemechanisms, for instance, by software running on computer equipment,hardware (e.g., chip-implemented logic functionality), etc., and/or anycombination thereof.

As to terminology, the phrase “configured to” encompasses any way thatany kind of physical and tangible functionality can be constructed toperform an identified operation. The functionality can be configured toperform an operation using, for instance, software running on computerequipment, hardware (e.g., chip-implemented logic functionality), etc.,and/or any combination thereof.

The term “logic” encompasses any physical and tangible functionality forperforming a task. For instance, each operation illustrated in theflowcharts corresponds to a logic component for performing thatoperation. An operation can be performed using, for instance, softwarerunning on computer equipment, hardware (e.g., chip-implemented logicfunctionality), etc., and/or any combination thereof. When implementedby computing equipment, a logic component represents an electricalcomponent that is a physical part of the computing system, howeverimplemented.

The following explanation may identify one or more features as“optional.” This type of statement is not to be interpreted as anexhaustive indication of features that may be considered optional; thatis, other features can be considered as optional, although notexplicitly identified in the text. Further, any description of a singleentity is not intended to preclude the use of plural such entities;similarly, a description of plural entities is not intended to precludethe use of a single entity. Further, while the description may explaincertain features as alternative ways of carrying out identifiedfunctions or implementing identified mechanisms, the features can alsobe combined together in any combination. Finally, the terms “exemplary”or “illustrative” refer to one implementation among potentially manyimplementations.

A. Overview of the System

FIG. 1 shows an overview of a system 102 for assisting a user 104 ininteracting with a physical and/or virtual space. A physical space maycorrespond, for example, an indoor space defined by the interior of oneor more buildings, or to an exterior space that exist outside of theinterior of buildings, or to some combination of indoor and outdoorspaces. A virtual space may correspond, for example, to a domain that ispopulated with virtual objects of any type or types. In some scenarios,a virtual space may be overlaid on a physical space through which theuser may physically move. In those cases, some virtual objects in thevirtual space may be assigned corresponding real positions in thephysical space (to be clarified below).

FIG. 1 broadly introduces illustrative components that may be used inthe system 102. The components are optional in the sense that aparticular implementation of the system 102 may omit one or more of theillustrated components. In addition, or alternatively, a particularimplementation of the system 102 may include additional components thatare not shown in FIG. 1.

During his or her exploration of the space(s), the user 104 may interactwith the system 102 via a user computing device 106 of any type(referred to as a simply a “user device” for brevity below) and aheadset 108, or just the headset 108 alone. The user device 106 maycorrespond to any type of portable computing device having any formfactor. For example, the user device 106 may correspond to a smartphone,a tablet-type computing device, a laptop computing device, anetbook-type computing device, a media consumption device (such as abook reader-type computing device or a music-playing computing device),a portable game device, a wearable computing device (such as eyewear,goggles, etc.), and so on. In other cases (not shown), a user 104 maycarry and utilize two or more user computing devices, such as asmartphone in combination with a tablet-type computing device.

The headset 108 may likewise correspond to any type of device fordelivering audio information to the user 104. In one case, for example,the headset 108 may deliver audio information using conventionalspeakers positioned over, or in proximity to, one or more ears of theuser 104. In another case, the headset 108 may deliver audio informationusing a bone conduction technique. In a bone conduction technique, theheadset 108 passes information to the user's eardrums via vibrationsimparted to the bones of the user's head. A headset that uses boneconduction may not block the ear canals of the user 104, and thus allowsthe user 104 to hear other (exterior) sounds produced by the spacesthrough which he or she navigates; this outcome may be desirable, insome cases, to increase the safety with which users interact with theirphysical environments, particularly for the case of sight-impaired usersand the case of users who are unfamiliar with their surroundings.

According to one general manner of use, the user 104 may interact withthe user device 106 to receive information regarding his or herinteraction with a space in primarily visual form. In addition, oralternatively, the user device 106 can deliver information in non-visualforms, such as by producing haptic feedback cues (e.g., vibration-basedcues). The user 104 may primarily interact with the headset 108 toreceive audio information regarding his or her interaction with a space.That audio information may include spoken information, other sounds,etc. The system 102 may also receive instructions from either the userdevice 106 or the headset 108, or both.

The user 104 may also optionally interact with the system 102 via one ormore traditionally stationary computing devices 110, such as a desktopcomputing device, a game console device, a set-top box device, and soon. For example, the user 104 may interact with the system 102 using theother user computing device 110 prior to a journey to create journeyinformation that defines a particular route through a space. The user104 may then load that journey information on the user device 106 and/orthe headset 108. Following the journey, the user 104 may again interactwith the system 102 via the other user computing device 110. Forexample, the system 102 may download information regarding a completedjourney to the other user computing device 110, allowing the user 104 toreview information regarding that journey at any time, for any purpose.

In one case, all functions associated with the system 102 are performedby processing functionality provided by the above-identified components,namely, the user device 106, the headset 108, and the other usercomputing device 110. In another case, one or more remote processingresources 112 may implement at least some aspects of the processingperformed by the system 102, such as those aspects which areparticularly computation-intensive in nature. For example, the remoteprocessing resources 112 may include functionality for creating newjourneys, modifying existing journeys, interpreting the user's spokeninstructions, and so on. In one case, the remote processing resources112 may correspond to one or more server computing devices, one or moredata stores, and so on.

The system 102 may use any combination of communication conduits 114 tocouple the above-described components together. For example, the userdevice 106 may interact with the headset 108 via any wirelesscommunication mechanism (e.g., using BLUETOOTH communication), and orany hardwired communication mechanism (e.g., via a USB connection,etc.). The user device 106 and the headset 108 may communicate withremote components of the system 102 via a cellular connection, a Wi-Ficonnection, a hardwired connection, etc., or any combination thereof.

Further, although not shown, the user device 106 may interact with anyremote position-determination systems for the purpose of determining theposition of the user device 106. The headset 108 may perform the samefunction. The remote position-determination mechanisms may correspond toany satellite-based position-determination system (e.g., a GPS system),terrestrial communication towers, and so on. Still further, the userdevice 106 and/or the headset 108 may also interact with local beaconsthrough any communication mechanism (e.g., via BLUETOOTH communication,Wi-Fi communication, etc.).

The remainder of this section (Section A) provides an overview of thefunctionality provided by the system 102. Later sections provideadditional details regarding individual components of the system.

In general, a user may use the system 102 as a way of enriching theuser's experience as the user moves through any type of space, familiaror unfamiliar. In one scenario, for instance, the user may use thesystem 102 as a guide in conducting a planned journey from a sourcelocation to a target location, subject to a desired timetable. In othercases, the user may use the system 102 to provide assistance inexploring a space in a more open-ended and spontaneous manner, e.g.,without a preplanned journey and/or timetable. For example, a user mayuse the system 102 to provide assistance as he or she wanders through anunfamiliar city, or meanders past the exhibits of a fair or museum; theuser may engage in this activity with no fixed itinerary, and may alterhis or her journey's course and objectives in a free-form andspontaneous manner. In yet another case, the user may use the system 102as a way of enlivening a familiar route through a familiar environment,and so on. In yet another case, a user may use the system in a hybridmanner of operation, e.g., in which some aspects of the user'sinteraction conform to a prepared plan and other aspects are moreopen-ended and spontaneous in nature.

Advancing to FIG. 2, this figure shows one example in which a user usesthe system 102 of FIG. 1 to take a preplanned journey. The user'sillustrative experience in conducting this journey will be describedbelow, as a way of introducing the reader to the types of functions thatthe system 102 may perform. Note that this example experience ispresented in the spirit of illustration, not limitation; as noted above,the system 102 can be applied in a wide variety of other contexts inwhich a user interacts with his or her environment, for any purpose.

In the non-limiting case of FIG. 2, assume that the user (a man namedJohn) has created a journey prior to embarking on the journey (although,again, this need not be the case). For example, assume that the user hascreated a route to take him from his residence in London to a doctor'sappointment in another part of the city. The user may have created ajourney for this task because he is unfamiliar with the part of the cityover which he is to travel. Or the user may have one or more handicapswhich present various mobility-related challenges. For example, the usermay have any form (and degree) of vision impairment. In this context,the user may have created the journey to assist him in navigating theroute, even though he may be familiar with the route. In another case,some entity other than the user may have created the journey on behalfof the user. In these cases, the planned journey is defined by journeyinformation, which describes all aspects of the journey.

FIG. 2 specifically represents the planned route 202 of the journey as asolid line. In some cases, the user may use the system 102 to generallyadhere to the planned route 202. In other cases, the user may divergefrom the planned route 202 for any reason, e.g., because he encountersan obstacle along the planned route 202, or he makes a spontaneousdecision to change the course of his travel for any reason. For example,in this merely illustrative case, the user has departed from the plannedroute 202 for the purpose of visiting a store 204 along the way, topurchase some item (e.g., a sandwich, etc.). FIG. 2 represents theactual route 206 of the user as a dashed line.

The planned route 202 in the illustrated case of FIG. 2 is defined by aseries of transitional points (w₁, w₂, w₃, w₄, and w₅) or stations,referred to as waypoints herein. For example, the waypoint w₁ maycorrespond to the starting point of the user's journey, while thewaypoint w₅ may correspond to the destination of the user's journey. Thewaypoints w₂ and w₃ may correspond to two intersections; at each suchintersection, the user transitions from a first road to a second road.The waypoint w₄ may correspond to any station at which a user may changehis mode of transportation. For example, the user may travel to waypointw₄ on foot. The waypoint w₄ may correspond to a shuttle station at whichthe user waits for the arrival of a shuttle 208, which arrives per apredetermined schedule. The user may then continue to the waypoint w₅ onthe shuttle 208. In other cases, the mode of transportation maycorrespond to a train, subway bus, tram, or any mode of privatetransportation (e.g., private automobile, bicycle, etc.).

Hence, the planned route may be conceptualized as having a series ofsegments (s₁, s₂, s₃, and s₄). The example of FIG. 2 is a simplifiedcase. In other cases, a journey may include many more waypoints andassociated segments. And that journey may encompass any combination ofmodes of transportation. In yet other cases, a journey may be lesscomplex than the one shown in FIG. 2, e.g., including only a beginningand ending waypoint. And, to repeat, in other cases, the journey may notbe defined in advance.

A context, as the term is used herein, generally refers to thecircumstance that confronts a user at any given time while conductingthe journey or otherwise interacting with the environment. Thecircumstance, in turn, is governed by at least features of theenvironment with which the user may wish to interact, the current timeof day (and day of week), etc., and the goals that confront the user atthe current time, etc. For example, at a time t₁, the user is traversingthe segment s₁ in an attempt to travel from the waypoint w₁ to thewaypoint w₂. The context c₁ of the user's journey is therefore defined,at least in part, by the user's current location along the plannedjourney, and the user's effort to reach the waypoint w₂ over the plannedsegment s₂. The user's context at other times will differ depending onthe environment that confronts the user at those times, coupled with theusers' respective local goals at that time.

With the above preliminary description of the planned route 202 and theactual route 206, now consider an illustrative user experience as theuser travels from waypoint w₁ to waypoint w₅. Assume, in one example,that the user carries at least the user device 106, and wears theheadset 108 (shown in FIG. 1). The user device 106 is referred to belowas a smartphone to simplify reference to this device, although the userdevice 106 can include any type of device mentioned above with referenceto FIG. 1. In yet other cases, the user may navigate with just the useof the headset 108 (that is, by eliminating the use of the smartphone).

As a general principle, the system 102 exposes the user to relevantinformation at appropriate junctures along the path of the user, as theuser conducts his journey or otherwise interacts with the space.Instances of the information that may be automatically presented to theuser in audible, visual, and/or haptic form are referred to herein asitems of interest (IOIs). To perform this function, the system 102determines the current context of the user at each moment of time. Forinstance, the system 102 senses the location and orientation (andoptionally, the motion) of the user at each particular time. The system102 can use any technology or combination of technologies to performthis task, examples of which are provided below in the context of thedescription of FIG. 3. The system 102 then identifies IOIs that arerelevant to the user's current context. In some cases, for instance, thesystem 102 can determine that an IOI is relevant to the user's currentcontext because the user is within a prescribed distance of a physicalobject (or region) that is associated with the IOI, where that distanceis defined and stored in advance. The system 102 then deliversinformation regarding those IOIs to the user. As will be described ingreater detail below, the user can also manually explore future contextswith which he will (or may) be confronted, at later times in thejourney, allowing the user to prepare for those situations.

The system 102 can provide the IOIs based on information extracted fromvarious sources. For instance, the system 102 can determine thelocations of roads, natural features, etc. from publically-available mapinformation (such as map information provided by the BING map serviceprovided by MICROSOFT Corporation of Redmond, Wash.). The system 102 candetermine the locations of public and private entities (e.g., stores,government buildings, etc.) from any published directory information.The system 102 can determine the occurrence of relevant events fromvarious published sources, such as public transportation scheduleinformation, public safety information. The system 102 can determineuser-related information from various services, such as one or moreonline social networking applications, one or more calendarapplications, etc., and so on.

Each IOI pertains to a particular topic or experiential focus. The IOIscan be categorized in different ways, along different explanatorydimensions. For instance, IOIs of a first class directly map to physicalobjects or physical events in the space through which the user is movingor otherwise interacting. For example, an IOI of this type maycorrespond to a store that lies in proximity to the user, or an openmanhole that lies in front of the user, or a next waypoint that the useris within a prescribed distance of reaching, etc. IOIs of a second classof IOIs do not necessarily have a direct physical counterpart in theenvironment. For example, an IOI of this type may correspond to anadvertisement that plays when the user approaches a bus stop. Theadvertisement may be associated with a space around the bus stop, butotherwise is not a description of the bus stop per se.

Another “virtual-type” IOI may correspond to a news headline that isbrought to the user's attention as he approaches a coffee stand. Thesystem 102 may present information regarding that IOI to the user basedon the premise that a user may wish to consume that IOI while drinking acup of coffee. Another virtual-type IOI may correspond to a weatherreport that is delivered to the user as the user leaves a subwaystation. The system 102 may present information regarding that IOI tothe user based on the premise that the user may wish to prepare for theweather which he is about to confront upon leaving the station. Anothervirtual-type IOI may correspond to a message retrieved from a socialnetwork application as the user approaches his or her personalresidence. The system 102 may present information regarding that IOI tothe user based on the premise that the user may wish to catch up on anymessages sent by family members or friends, prior to entering his home.Many other scenarios are envisioned in which the user receives some typeof information (and/or opportunity to interact with functionality) thatis germane to the user's present circumstance, yet might not serve todescribe an actual object or event in the user's immediate vicinity.

Different IOIs (of either the first “real” class or second “virtual”class described above) can also be categorized based on their roles thatthey serve in the exploration of space. For example, the IOIs may beclassified along this dimension as warning IOIs, journey IOIs,contextual IOIs, and available-information IOIs, etc. (although suchcategories may not be, strictly speaking, mutually exclusive). Thesystem 102 sends information regarding warning IOIs to alert the user tooccurrences that may affect the safety of the user during the journey(such as an indication that there is an open manhole in front of theuser). Warning messages are sharp and to the point. The system 102 sendsinformation regarding journey IOIs to alert the user to occurrences thatmay affect the progress of the user's journey (such as an indicationthat a next waypoint is approaching, or that a bus will be late). Thesystem 102 sends information regarding contextual IOIs to the user toalert the user to objects and events in the vicinity of the user(whether real or virtual) that may interest the user (such as anindication that a coffee shop is ahead of the user on his journey). Thesystem 102 alerts the user to the existence of available-informationIOIs, without automatically delivering them. A user may then choose toreceive this information in an on-demand manner, or ignore it. Forexample, a user may access the available-information by tapping on anappropriate function button on the headset 108 or through an informationmenu provided by the user device 106, or using the equivalent voicecommand.

The system can deliver information regarding each IOI in any manner. Forexample, for at least some IOIs, the system can deliver the IOI bysending a telltale sound, followed by a spoken message. The preliminarysound enables the user to “tune in” to hear the spoken announcement,that is, by directing attention to the spoken announcement. The soundalso alerts the user as to the type of information that is to follow, sothat the user is better able to decide whether he or she will devoteattention to it. For example, users may choose to give heightenedattention to journey information, but devote less attention tocontextual information (e.g., regarding a store in the user's vicinity,or a promotional offer pertaining to the user's current circumstance).

At any point in the journey, the user can make a request to hear theinformation regarding one or more journey IOIs (and/or other types ofIOIs) that were most recently read aloud. That request can be made invarious ways to be described below. If a warning has been read aloudwithin the last 30 seconds, then the system 102 will respond to theuser's action by also repeating the warning, followed by repeating thejourney information. In some cases, the user may request the system 102to repeat information, but this information is no longer available. Thesystem 102 can play an appropriate sound to alert the user to thissituation, following by a spoken message, “information is unavailable atthis time,” or the like.

As another general feature, the system 102 may automatically presentinformation regarding some IOIs at times at which the user is notexpected to be overwhelmed with other sensory information from theenvironment. For example, the system 102 may refrain from sending theuser messages regarding the planned route 202 until the user reaches aprescribed “quiet zone” along the planned route, at which the user cansafely and enjoyably consume the information. The system 102 can store,in advance, the locations of regions that are considered “quiet zones.”

Now more closely consider the user's particular experience whilenavigating over the space shown in FIG. 2. Assume that, at time t₁, theuser is attempting to reach the second waypoint w₂. After sensing thelocation (and heading) of the user, the system 102 can send the userdirections (in visual and/or audible form) that assist the user inreaching the second waypoint w₂. Further, the system 102 can send theuser information regarding the second waypoint w₂ itself when the useris within a prescribed distance from the second waypoint w₂. This willenable the user to make suitable preparations for any change in coursethat the second waypoint w₂ may entail. The above-described informationmay be formulated primarily as journey IOIs.

In certain situations, the user's actual journey may depart from theplanned journey, e.g., with respect to the route that is actually takenand/or the timing of the user's traversal of the space. To address thosesituations, the system 102 can automatically (or in an on-demand manner)determine whether the user's current situation will impact any remainingparts of the user's journey. For example, the system 102 can update theuser's estimated time of arrival at a bus station and then determinewhether the user will continue to arrive on time to catch apreviously-identified bus or shuttle. If the user's current situationimpacts the user's journey in this manner, the system 102 canautomatically re-generate information that is used to assist the user innavigating within the space. That information may be expressed as a newset of IOIs. For example, the system 102 can advise the user to take alater bus or shuttle (and can automatically make appropriatereservations, send appropriate notifications, and/or make otherarrangements). Further, as noted above, the user may explore the updatedinformation in any manner, e.g., by expressly requesting informationregarding a waypoint to be encountered in the future, etc.

FIG. 2 also generally indicates that, at time t₁, the system 102automatically informs the user of the existence of various items ofinterest (IOIs) of a contextual nature (that is, in the terminology setforth above, “contextual IOIs”). For example, an illustrative contextualIOI may correspond to a store, a restaurant, a government office, anatural landmark, etc. In one case, the system 102 may present acontextual IOI for the user's consideration only if it is within aprescribed distance of the user the present time, and the user has notalready passed it by.

The user may customize the behavior of the system 102 in any manner(with respect to contextual IOIs and/or any other type of IOI). Forexample, the user can specify the types of prompts that he wishes toreceive along his route, e.g., by indicating he would like to receiveinformation regarding a first type of store, but not a second type ofstore. The user may also specify the timing at which he would like toreceive the information. For example, the user can specify the maximumdistance (from his current location) that should be considered whennotifying him of the existence of contextual IOIs. The user may makethese settings prior to embarking on the journey. In addition, the usermay dynamically change the type and quantity of information delivered bythe system over the course of the journey, e.g., by dialing back on theamount of information that is automatically provided by the system. Theuser may opt to reduce the amount of information because he finds itunnecessary or distracting at a particular juncture along the journey.

The system 102 can present information regarding contextual IOIs in anyapplication-specific manner. For example, in one case, the system 102may announce a collection of contextual IOIs in the order in which theyappear in front of the user, from left to right or from right to left(e.g., by essentially forming an angular sweep, with the user positionedat the origin of the sweep), or from front to back (in terms of distancefrom the user), or back to front, etc. As noted above, in oneimplementation, the system 102 can precede each announcement with atelltale sound that alerts the user that information regarding acontextual IOI is about to follow. The system 102 may then describe thecontextual IOI, e.g., by providing audio information which announces,“Jane's Coffee Shop, 150 feet.” The system 102 may alternatively providedifferent preliminary sounds associated with different types ofestablishments, such as by providing a first type of preliminary soundfor restaurants, and a second type of sound for bus stops, etc.

In some cases, there may be a large quantity of contextual IOIs withinthe vicinity of the user at a current time. To cut back on the amount ofinformation imparted to the user, the system 102 may consolidate thisinformation into one or more summary messages, such as by announcing,“Restaurants, generally at 100 feet.” The same holds true for any typeof IOI. In general, the system 102 can determine whether a group ofindividual IOIs to be delivered to the user at a given time have atleast one common characteristic. The system 102 can then provide asummary message which announces the group of IOIs as a block or set,rather than individually. In the specific example noted above, thesystem 102 determines that a group of contextual IOIs are located in thevicinity of the user and that these IOIs pertain to the same type ofestablishment. In another example, the system 102 may provide thesummary warning IOI, “Numerous large potholes in the next 100 yards.”

The system 102 may use other rules to provide announcements in anintelligent manner. For example, some IOIs are relevant only if the useris in close proximity to these IOIs. For example, a user may beinterested in a park bench only if the user is within a few feet of thisobject. Hence, the system 102 may announce the presence of these kindsof objects only when the user is relatively close to these objects. Buthere again, the user may customize the behavior of the system in thisregard.

According to another illustrative feature, the system 102 may usethree-dimensional sounds to announce the presence of some types of IOIs,such as some types of contextual IOIs. A three-dimensional sound refersto a sound that the user perceives as emanating from a particularlocation (or locations) within physical space. In reality, however, theaudio information is presented to the user via the headset 108 and hasno physical origin within the environment. As will be described below inSection D, the system 102 can achieve the above result through the useof Head-Related Transfer Functions (HRTFs), in conjunction withdifferent types of wideband audio sounds. In other implementations, thesystem 102 can use other techniques and technologies to create athree-dimensional audio effect (instead of HRTFs or in addition toHRTFs), such as Ambiophonics, Ambisonics, wave field synthesis etc.

For example, at the context c₁ (at time t₁), the system 102 may send aseries of directional announcements to the user which identify at leastthree contextual IOIs (IOI₁, IOI₂, and IOI₃). The system 102, forinstance, can announce the presence of contextual IOI₁ by sending apreliminary sound to the user which the user perceives as emanating froma particular location in space at which the entity associated with thecontextual IOI is physically located. That is, if the contextual IOIcorresponds to Jane's Coffee Shop, the message would direct the user'sattention to the actual location of Jane's Coffee Shop. The system 102can also provide the description of this establishment using athree-dimensional sound, such as by announcing “Jane's Coffee Shop, 150feet” in a directional manner, that is, using three-dimensional spokeninformation.

In addition, or alternatively, the system 102 may send the userinformation regarding one or more information-available IOIs. Asexplained above, the system 102 can perform this operation by sending amessage which generally alerts the user to the existence of informationthat may pertain to the user's current context, but without immediatelyannouncing the particular substance of that information. The user maythen manually ask the system 102 to receive the information, e.g., bymaking a “more information” instruction. Or the user may ignore theinformation.

All of the above information-delivery features contribute to the goal ofsurfacing useful information to the user as the user traverses a space,without overwhelming the user with too much information.

As another feature, at any juncture along the user's path, the system102 can compare the user's actual direction of travel with a desireddirection of travel. The system 102 can determine the user's actualdirection of travel using any of the mechanisms described below inconnection with FIG. 3. The system 102 can determine the desireddirection by determining where the user is expected to be headed at thepresent time. The system 102 can then determine deviation informationwhich determines an extent to which the actual direction of the userdeviates from the desired direction. The system 102 can then send theuser information which attempts to steer the user along a desired path.

For example, consider the user at time t₂. At this juncture, the user isattempting to reach waypoint w₃. Assume that the user's actual directionis represented by arrow 210, and the user's desired direction isrepresented by arrow 212. The system 102 will send the user instructionsthat attempt to steer the user away from his current errant direction,towards the desired direction.

In one implementation, the system 102 accomplishes the above goal usinga three-dimensional beat sound or other type of periodic sound. Thethree-dimensional beat sound is any type of repeating sound (such as aclicking sound) that is perceived by the user as originating from aparticular location (or locations) in the physical space. In the case ofFIG. 2, at the time t₂, the system 102 will deliver a three-dimensionalbeat sound that appears to originate to the left of the user along hisdirection of travel. This will advise the user to the fact that: (a) heis headed in a non-optimal direction; and (b) he should turn slightlyleft to achieve a more desirable trajectory.

The system 102 can modulate the beat sound to achieve other effects. Forexample, the system 102 can change the tone and/or periodicity and/orvolume (and/or some other aspect) of the beat sound depending on theextent to which the user is currently headed in a non-optimal direction.The user will interpret the beat sound as an indication of the extent towhich he is headed in the wrong direction.

So far, the description has mostly emphasized the ability of the system102 to automatically deliver information to the user at appropriatejunctures along the user's path. In this mode of operation, the user'sencounter with the environment may constitute the events which triggerthe delivery of information, e.g., in the form of the above-describeddifferent types of IOIs. In addition, at any time, the user may manuallyinteract with either the smartphone or the headset 108 to manuallyexplore his environment and to obtain information regarding any of theIOIs described above. For example, in a first manner of interaction, theuser may tap on a touch-sensitive surface of the smartphone at any giventime along the journey. In response to a single tap, the system 102 willannounce top-level information regarding the user's present context. Forexample, at time t₁, the system may respond to a single tap byannouncing that the user is headed to waypoint w₂, which may beconsidered a journey IOI. In response to a double tap, the system 102will provide more detailed information regarding the present context. Inresponse to a triple tap, the system 102 will provide instructions whichallow the user to interact with the smartphone, e.g., for the purpose ofobtaining additional information regarding the context, and to invokevarious functions pertaining to the current context.

As another feature, at any juncture, the user may interact with thesmartphone to activate one or more menus that are relevant to the userat a given time. For example, at time t₁, the user may perform atap-and-hold gesture on the surface of the smartphone. In response, thesystem 102 may activate a menu associated with the user's presentcontext. The user may then interact with the smartphone to explore theimmediately presented menu, or navigate to any other menu.

More specifically, as will be explained in detail in Section C below,the system 102 may represent a collection of menus that are accessiblethrough a set of workspaces. Each workspace has a fixed positionalrelationship with respect to each other workspace. The user may accessdesired information and/or functionality by navigating to an appropriateworkspace and associated menu.

According to one feature, the user may perform all of the abovescreen-interaction tasks with a single hand, e.g., using the thumb ofthe hand that holds the smartphone. The system 102 can also provideaudible and/or haptic feedback cues as the user interacts with thesmartphone's touch-sensitive surface. Collectively, all of thesefeatures reduce to extent to which the user needs to divert hisattention from the environment while interacting with the system 102.For example, the user may keep his eyes on the road on which he iswalking while interacting with the smartphone. A person havingimpairment of his sight can also successfully interact with the system102 due to the above-summarized non-visual characteristics.

As will be described in Subsection C.3, the user may also manuallyinteract with the system 102 via voice instructions. In addition, oralternatively, the user may manually interact with the system 102 viainput mechanisms provided by the headset 108 (as described in SectionB). The system 102 may provide yet other mechanisms by which the usermay manually interact with the system 102.

The user may also invoke special modes for use in exploring hisimmediate environment. For example, in response to activating an exploremode, the system 102 can determine the current focus of attention of theuser, which may correspond to the direction at which the user ispresumed to be looking at the current time (which, in turn, may bedetermined based on one or more orientation determination mechanisms).The system 102 can then determine the contextual IOIs that areencompassed by (or otherwise associated with) a subspace formed aroundthe user's direction of attention. The system 102 can then read offthese contextual IOIs by announcing these contextual IOIs usingthree-dimensional sounds, and/or by sending visual messages forpresentation on the user's smartphone, etc.

Some of these contextual IOIs may pertain to real objects in theenvironments having respective real positions within the subspace. TheseIOIs serve principally to identify or mark the locations of thesephysical entities in physical space. Other contextual IOIs may bevirtual in nature, in that they pertain to the subspace (given theuser's current context), but may not directly describe objects in thatsubspace. In other words, these other contextual IOIs convey informationand/or an experience that relates to the user's current context, beyondthat of identifying the locations of physical entities.

To cite one example, a contextual IOI of the “virtual” variety maycorrespond to a message, “Remember our fallen troops” as the userapproaches a military barracks. That message is related to the barracksbut cannot be accurately said to simply mark the location of thebarracks. The intent of that message is to create a mental associationto enrich the user's experience as he approaches the barracks.Alternatively, the contextual IOI in that circumstance may correspond toa song or an excerpt from an inspirational speech or a personal messagethat has particular meaning to the user (as previously specified andloaded into the system 102 by the user).

The same is true for other types of IOIs (other than contextual IOIs).That is, some IOIs serve mainly as labeling tags, while other IOIs areintended to stimulate cognitive associations, memories, emotions, etc.The latter group of IOIs is referred to as “virtual” herein in the sensethat they pertain to a realm of associations, experiences, meanings,etc. that is not a surface-level transcription of events and objects inthe environment. Such IOIs could alternatively be referred to asinferential, suggestive, relational, etc. IOIs.

According to another feature, in response to activating an orientationmode, the system 102 can perform a complete 360 scan around the user toidentify all items of interest that are associated with a prescribeddistance from the user. The system 102 can also perform this 360 degreescan for successive levels in the vertical dimension, e.g., to determinestores provided on different levels of a mall complex or the like. Theuser can customize the behavior of the explore mode and the orientationmode in any manner described above, e.g., by changing the types of IOIsthat are identified, the dimensions of the space that is searched forthe presence of the IOIs, and so on. In addition, the user may interactwith the system 102 to govern the manner in which the system 102 readsoff the IOIs.

Now assume that, at time t₃, the user spontaneously decides to veer offfrom the planned route 202 to visit the store 204, e.g., to purchase asandwich. When in the store, the context (c₃) of the user at this timepertains to a store environment. Hence, the system 102 may perform thesame functions as described above, but now in the context of the indoorenvironment of the store. For example, the system 102 can automaticallydetermine contextual IOIs as the user traverses an isle of the store204, and announce those contextual IOIs to the user. For example, uponapproaching the dairy section of the store 204, the user may receive amessage that reads, “Milk, cheese, and yogurt, 20 feet ahead.” Thesystem 102 may send progressively more detailed information as the userdraws closer to products that may interest him. Again, some of thecontextual IOIs can have a less direct correspondence with physicalobjects in the store, e.g., as in a message that is delivered in thesoup section, which alerts the users of the presence of high sodium inmany soups.

The user may also manually interact with the system 102 within the storeenvironment in any manner. For example, the user may manually exploredifferent menus associated with different products. The user may alsouse the smartphone to perform various transactions in the storeenvironment, such as purchasing an item, researching an item, etc.

In some implementations, the system 102 can determine the location ofthe user within the store 204 by determining whether the user is withinthe range of one of a set of beacons, which have been placed (inadvance) at different locations within the store 204. As will bedescribed in Section E, the beacons may have non-overlapping ranges.

Upon leaving the store 204, the system 102 may recalculate the user'sjourney to lead the user back to the planned route 202. For instance,the system 102 may provide the user with instructions that allow theuser to reach the waypoint w₄, associated with a shuttle stand. Uponreaching that waypoint, the system 102 may then deliver information thatis relevant to the user at this juncture, such as by announcing theexpected time of arrival of the user's shuttle 208. That information maybe delivered as one or more journey IOIs.

The system 102 may continue to provide services to the user while he istravelling on the shuttle 208. For example, the system 102 may notifythe user of the expected time of arrival at the ultimate destination,i.e., waypoint w₅ (the user's doctor's office). The system 102 can alsoprovide other messages that may be of use to the user, depending on thenature of the public (or private) transportation on which the user istraveling. For example, when riding on a bus and approaching a finaldestination, the system 102 can alert the user to the existence of ahigh curb that he is expected to encounter upon exiting the bus.Further, the system 102 can, with the user's permission, alert thedriver of the bus that a person requiring assistance will bedisembarking at an upcoming bus stop.

In summary, throughout the user's journey, the user may receive a largequantity of information in audible form, e.g., in the form of spokenmessages, other sounds (three-dimensional sounds andnon-three-dimensional sounds), etc. The system 102 may use varioustechniques to manage the presentation of this information, some of whichhave been already mentioned above (such as the ability to dial back ordial up on the quantity of information that is delivered). This featureallows the user to receive the most relevant information in a timelymanner, without overwhelming the user with too much information.

For example, the system 102 may play back sounds during a journeysubject to different rules to address the situation in which thedelivery of one sound potentially interferes with the delivery ofanother sound. According to one illustrative rule, the system 102 willplay the beat sound in a continuous loop to steer the user in a desireddirection, e.g., while walking. The system 102, however, may temporallydisable this sound (or reduce the volume of this sound compared to anormal state of this sound) when any other sound is being played. Thisenables the user to hear the other sounds without interference from thebeat sound, which is considered a low priority sound.

According to another illustrative rule, the system 102 mayunconditionally play sounds that represent interface-related events,such as flick gestures which change which menu or context is presentedto the user. To avoid overloading the user with too much audioinformation, these types of sounds may be designed to be short anddistinct. A user may control the playback of these types of cues, atleast to some extent, by temporally suspending his or her interactionwith the system 102 (because no interaction cue will be produced if theuser is not actively interacting with the system 102).

According to additional illustrative rules, the system 102 canprioritize navigational sounds by assigning the highest priority levelto warning sounds (e.g., for warning IOIs), the next highest prioritylevel to journey information (e.g., for journey IOIs), and the nexthighest level to contextual information (e.g., for any type ofcontextual IOIs). In some cases, the system 102 will delay the deliveryof information, e.g., because more critical information is being playedback. Further, in some cases, a delayed message will no longer berelevant by the time that the system 102 is capable of presenting it(e.g., because the user has moved to a new context in which theinformation is no longer relevant); if so, the system 102 may refrainfrom presenting that information.

In conclusion, the above-described scenario is also useful inhighlighting some of the advantageous technical effects of the system102. Generally, the system 102 allows any user to receive guidance thatserves different but related goals. First, the system 102 attempts toexpose the user to the most useful information at any given time alonghis or her journey, thus empowering the user to more effectivelynavigate within his or her environment, or to achieve other objectives.Second, the system 102 enriches the user's experience of the environmentbeyond providing navigational assistance, allowing the user to learn newinformation about the environment that may not be immediately apparentto the user without the use of the system 102; in this regard, thesystem 102 allows the user to metaphorically delve beneath the surfaceof the environment to understand formerly hidden aspects and connectionswhich pertain to the environment 102. Third, the system 102 attempts toprovide this useful information to the user in a manner which minimizesthe distractions placed on the user. The third goal is useful to providea more enjoyable and useful experience to the user, e.g., by allowingthe user to maintain primary focus on his or her interaction with the“real world,” not on the tools which he or she is using to interact withthe real world. Stated in the negative, the third goal attempts toreduce the anxiety that may occur by asking the user to interact with acumbersome and complex tool, to which the user would be expected todevote significant attention. The third goal also allows the user toefficiently and quickly access desired information in a safe manner,without being overwhelmed at any given time with too much information.

A number of technical features contribute to the above-summarized goals,particularly with respect to the third goal. The features include, butare not limited to: a) the use of a one-handed interaction experience;b) the use of a user-friendly menu structure that accommodates gesturesthat can be performed on a touch-sensitive surface in alocation-agnostic manner (to be described below); c) the use of auser-friendly and easy-to-learn workspace structure which providesaccess to a “safe” home workspace (to be described below); d) the use ofmultiple mechanisms to enter commands (e.g., via the headset 108, userdevice 106, voice recognition, etc.); e) the use of audio informationand/or haptic cues to convey information without unduly disrupting theuser's focus on the journey; f) the use of three-dimensional andnon-three-dimensional sounds to help steer the user in a desireddirection without inundating the user with complex instructions, or toalert the user to the location of IOIs, and so on.

The above advantages apply to any user of the system 102. The system 102may also be successfully used by people with any type of condition thatimpairs their ability to make journeys. These users may include usershaving partial or full loss of sight, users with cognitive or otherpsychological impairments, users having mobility-related handicaps, andso on. For these users, the system 102 acts as a virtual guide dog,assisting the user in each stage of their journey in a safe manner, orotherwise assisting the user in interacting with their environment,whatever the user's goal happens to be at the moment. For these users,in addition to the above-summarized general benefits, the system 102also allows the user to access information and guidance that would nototherwise be available to them, thus potentially improving the mobility,confidence, and general quality of life of these users.

Advancing now to FIG. 3, this figure shows a high-level overview ofcomputing functionality that may be used to implement the system 102 ofFIG. 1. The functionality of FIG. 3 is shown in a device-agnosticmanner. In an actual implementation, the functionality may be allocated,for instance, to any of the components introduced in FIG. 1, or anycombination of these components. For instance, FIG. 3 shows that thefunctionality includes a space interaction (SI) module 302. The SImodule 302 performs all (or most) of the functions described withrespect to the scenario of FIG. 2. Some parts of the SI module 302 maybe implemented by the user device 106, while other parts of the SImodule 302 may be implemented by processing components located on theheadset 108. In addition, or alternatively, some parts of the SI module302 may be performed by the remote processing resources 112.

The SI module 302 may receive input information from any combination ofinput mechanisms 304, and may provide its output information forpresentation on any output mechanisms 306. For example, the inputmechanisms 304 may include one or more orientation determinationmechanisms 308, and/or one or more motion determination mechanisms 310,and/or one or more position determination mechanisms 312, and so on.

The orientation determination mechanism(s) 308 determine the orientationof the device which incorporates these mechanism(s) 308. For example, ifhoused by the user device 106, the orientation determinationmechanism(s) 308 determine the three-dimensional orientation of thisuser device 106. If housed by the handset 108, the orientationdetermination mechanism(s) determine the three-dimensional orientationof the headset 108. More generally stated, the orientation determinationmechanism(s) 308 may determine the direction that the user is pointinghis smartphone or turning his or her head (on which the headset 108 isplaced). The motion determination mechanism(s) 310 determine the natureand degree of movement of the device which incorporates thesemechanism(s) 310. The position determination mechanism(s) 312 determinethe absolute and/or relative position of the device which incorporatesthese mechanism(s) 312.

The mechanisms (308, 310, 312) can be implemented using any combinationof sensors, including but not limited to: magnetometers, accelerometers,gyroscopic sensors, gravity-based sensors, torque sensors, straingauges, flex sensors, optical encoder mechanisms, and so on. Inaddition, some of the mechanisms (308, 310, 312) may receive signalsfrom external systems or sources. For example, the mechanisms (308, 310,312) may include a sensor for determining the position of a device basedon signals received from a satellite-based navigation system (e.g., aGlobal Positioning System (GPS) system). In addition, or alternatively,the mechanisms (308, 310, 312) may include functionality for determiningthe position of a device by performing triangulation and/or otherprocessing based on signals received from plural external sources, suchas signals received from plural radio towers and/or localizeddirectional antennas, etc. In addition, or alternatively, the mechanisms(308, 310, 312) can include functionality for determining the positionof a device using a dead reckoning technique. In addition, oralternatively, the mechanisms (308, 310, 312) can include functionalityfor determining the position of a device by processing information fromlocal beacons (e.g., Wi-Fi and/or BLUETOOTH beacons, etc.), and so on.

The input mechanisms 304 may also include any combination of manualinput mechanisms 314. These mechanisms can include any of: key inputmechanisms, touch-sensitive input mechanisms (e.g., a touch-sensitivescreen 314′), joysticks, microphones (e.g., for receiving voiceinstructions), video cameras and/or depth cameras (e.g., for receivingfree space gestures), and so on. For example, in the case of FIG. 1, theuser device 106 may use a touch-sensitive display screen as its primarymanner of interacting with the user. For instance, without limitation,that touch-sensitive display screen may incorporate a capacitivetouchscreen mechanism that determines when a user touches and/or hoversabove the screen. The user device 106 may also include a camera, amicrophone, etc. The headset 108 may include a microphone (for receivingvoice instructions) together with one or more dedicated inputmechanisms, e.g., as implemented as buttons on the side of the headset108 (to be described in greater detail in the next section).

The output mechanisms 306 can incorporate one or more audio outputmechanisms 316, one or more display output mechanisms 318, one or morehaptic output mechanisms 320, and so on. For example, the audio outputmechanism(s) 316 can correspond to conventional speakers of any type. Inaddition or alternatively, the audio output mechanism(s) 316 canincorporate bone conducting audio devices (e.g., as provided by theheadset 108), such as bone conducting transducers produced byAFTERSHOKZ, LLC, of Syracuse, N.Y. The display output mechanism(s) 318may correspond, for instance, to a LCD type display (e.g., as providedby the user device 106). The haptic output mechanism(s) 320 maycorrespond, for instance, to a vibration-producing mechanism (e.g., asprovided by the user device 106 and/or the headset 108, etc.). Avibration-producing mechanism may achieve a vibration effect using arotating off-balance weight, and/or by some other mechanism(s).

The SI module 302 may also interact with remote functionality 322 thatmay be considered external to the SI module 302 per se. For example, theSI module 302 may interact with a search engine 324 for the purpose ofconducting a search. For example, the search engine 324 may correspondto the BING search engine provided by MICROSOFT Corporation of Redmond,Wash. In addition, or alternatively, the SI module 302 may interact witha journey computation engine 326 for the purpose of generating a newjourney and/or modifying an existing journey. In addition, oralternatively, the SI module 302 may interact with a speech processingengine 328 to interpret the spoken instructions made by the user. Forexample, the speech processing engine 328 may correspond to the CORTANAsystem provided by MICROSOFT Corporation of Redmond, Wash. In othercases, one or more aspects of the remote functionality 322 may beincorporated in the SI module 302 as native resources.

In addition, the SI module 302 may interact with any externalinformation provided in one or more data stores 330. For example, theexternal information may provide publically accessible map information,transportation schedule information, alert information, business andpersonal directory information, social network information, calendarinformation, etc., and so on. In some cases, the SI module 302 mayinteract with external sources (e.g., external websites) usingapplication programming interfaces (APIs) provided by those externalsources.

Now referring to FIG. 4, this figure shows one implementation of the SImodule 302, which was introduced above. From a high-level standpoint,the SI module 302 may include (or may be conceptualized as including) aplurality of sub-components that perform different respective functions.Further, some sub-components may rely on the results generated by othersub-components. An application interface module (AIM) 402 allows theuser to interact with any of the sub-components. For example, theapplication interface module 402 may provide menu functionality whichexposes various functions provided by the sub-components.

Generally referring to the sub-components from top to bottom, the SImodule 302 may include various data stores for storing information thatmay be used by the SI module 302 in performing its functions. Forexample, a data store 404 may store information which defines one ormore journeys. For example, the journey information may describe thewaypoints in the journey, and/or any other information regarding thejourney. A data store 406 may store search results provided by a searchengine; those results may be produced, upon direction of the user,during the course of the journey or a user's more general interactionwith a space. A data store 408 may store a history of tabs created bythe user in the course of the user's interaction with the SI module 302.A tab generally corresponds to a bookmark to a menu or other item ofinformation and/or functionality and/or option. A user may create a tabwhen he or she visits that menu or other item or information and/orfunctionality and/or option; in one implementation, the data store 408initially contains no tabs when a user embarks on a journey and has notyet started to interact with the system 102. The system 102 may use anymanner of representing a collection of tabs, e.g., as a list of tabs, aradial menu of tabs, etc.

The SI module 302 can also provide various supporting modules 410 thatperform any type of support services. For example, a setting module mayallow a user to assign a value to any parameter which affects theoperation of the system 102. A data store 412 may store all suchsettings. The supporting modules 410 may also include a portal forinteracting with an external search engine, to provide search results.Or the supporting modules 410 may include a natively-provided searchengine.

A sound generation module 414 performs various operations relating tothe generation of sounds. For example, the sound generation module 414may play particular sounds when various triggering circumstances areencountered. Some triggering circumstances correspond to actions madethe user when interacting with the application interface module 402.Other triggering circumstances correspond to changes in the state of thesystem 102 that are not directly caused by the user's interaction withthe system 102. Other trigger circumstances correspond to events whichoccur during the course of a journey (or more generally, the user'sinteraction with a space), and so on. A data store 416 may store fileswhich, when played back, produce the desired sounds. Subsection C.2(below) provides additional information regarding different types ofsounds that may be generated, and the circumstances under which thesesounds are played.

Some sounds are non-three-dimensional or non-spatial in nature. Inaddition, the sound generation module 414 can produce three-dimensionalaudio information. As described above, the audio information isthree-dimensional in the sense that a user will perceive thisinformation as emanating from one or more locations within athree-dimensional physical or virtual environment. The sound generationmodule 414 may create these sounds by transforming original soundinformation using one or more Head-Related Transfer Functions (HRTFs).

Similarly, a haptic cue generation module 418 can produce differenttypes of haptic feedback experiences in different triggeringcircumstances. In one case, the haptic cue generation module 418produces signals which produce vibration cues, e.g., when delivered tothe user device 106, the headset 108, and/or some other device.

A path guidance module 420 uses the sound generation module 414 togenerate the above-described three-dimensional periodic (e.g., beat)sound. The purpose of this periodic sound is to guide the user in aparticular direction. The path guidance module 420 produces this effectby determining the current actual heading of the user, the desiredheading, and the difference between the actual and desired headings(corresponding to deviation information). Then the path guidance module420 then leverages the sound generation module 414 to produce anappropriate looping sound that is perceived by the user as originatingfrom a particular direction. That is, the sound generation module 414produces the looping sound based on the deviation information fed to itby the path guidance module 420. The user may respond to this sound bymoving in the desired direction. In another case, the beat sound may beperceived as travelling across a series of locations in physical space.The user may interpret this experience as an instruction to move in thedirection of the traveling sound.

A beacon-based guidance module 422 provides assistance to the user innavigating within an indoor and/or output space by means of thedetection of signals emitting from a collection of beacons having, inone illustrative implementation, respective non-overlapping ranges.Section E provides additional information regarding the operation of thebeacon-based guidance module 422. The beacon-based guidance module 422may consult beacon information provided in a data store 424. The beaconinformation may describe the codes associated with the beacons that havebeen placed in the environment and their respective locations within theenvironment.

A relevant information determination (RID) module 426 performs thegeneral function of determining relevant information to present to theuser at any given time. In context of the description of FIG. 2, the RIDmodule 426 determines different types of items of information (IOIs)that have a bearing on the user's current context. To perform this task,the RID module 426 receives various contextual inputs that define thecontext of the user at the present time. These contextual inputs maydescribe the current location of the user, the current heading of theuser, the current goals of the user, and so on. The contextual input mayalso describe the environment itself, such as objects and eventsassociated with the environment, both physical and virtual. Any suchinputs may be mined from map information, directory information, socialnetwork information, calendar information, etc. The contextual inputsmay also describe environmental factors which affect the user'sinteraction with the space, such as public transportation information,weather information, etc., as obtained from any source(s).

The RID module 426 operates by determining whether it is appropriate tonotify the user of any information at a given time, based on thecontextual inputs at the present time, and based on various rules(provided in a data store 428). The behavior of the RID module 426 mayalso be defined by one or more parameters set by the user and stored inthe data store 412. For example, the RID module 426 may determinewhether there are any contextual IOIs in proximity to the user at thepresent time, based on a depth range defined by the user. If suchcontextual IOIs exist, the RID module 426 can interact with the soundgeneration module 414 and/or menu functionality provided by theapplication interface module 402 to notify the user of these contextualIOIs.

An exploration module 430 and an orientation module 432 performrespective services that may be invoked in an on-demand manner by theuser. As described with reference to the scenario of FIG. 2, theexploration module 430 determines any contextual IOIs that areassociated with a subspace that lies in front of the user (which, inturn, may be determined by the position of the user's head, togetherwith a setting parameter which defines the depth of investigation, and asetting parameter which describes the span of the search space). Toperform this task, the exploration module 430 leverages the services ofthe RID module 426. The exploration module 430 then notifies the user ofthe contextual IOIs using three-dimensional sounds, displayed messages,etc. The orientation module 432 performs a similar task to theexploration module 430. But instead of investigating IOIs associatedwith a subspace that projects out in front of the user, the orientationmodule 432 can scan the entire three-dimensional space which existsaround the user at the present time.

FIG. 5 shows one implementation of the application interface module(AIM) 402. As noted above, the application interface module 402generally provides an interface that allows the user to interact withthe various sub-components described above, with reference to FIG. 4.

The application interface module 402 may include various components thatinterpret the input of the user, e.g., to determine the nature of aninstruction that the user is making. For example, a gestureinterpretation module 502 may determine a gesture that the user has madeby interacting with the touch-sensitive screen of the user device 106,or a free-space gesture, etc. The gesture interpretation module 502 mayperform this task by comparing the marks or touches or hovers (or otherbehavior) made by the user with a data store that identifies patternsassociated with known gestures. If the behavior of the user matches apattern associated with a particular gesture with a determined degree ofmatching confidence, then the gesture interpretation module 502 mayconclude that the user has made that gesture.

A voice interpretation module 504 may interpret spoken instructions bythe user, e.g., which may be received via a microphone on the userdevice 106 and/or the headset 108. In one case, the voice interpretationmodule 504 may correspond to a portal to the remote speech processingengine 328 (e.g., the CORTANA system). In another case, the voiceinterpretation module 504 may correspond to any type of nativefunctionality for interpreting spoken utterances. In any event, theagent which performs voice recognition can use any technology to performthis task, such as Hidden Markov Model-based technology, neural networkbased technology, etc.

A headset button interpretation module 506 interprets the manner inwhich the user interacts with the input mechanisms of the headset 108(to be described below). For example, in some cases, a set of buttonscan perform different functions depending on the manner in which theuser interacts with them, e.g., depending on whether the user touches abutton but does not press it, or based on whether the user touches andreleases the button one or more times, or based on whether the userpresses and holds the button, etc. The headset button interpretationmodule 506 maps the user's behavior to a particular instruction.

An action-taking module 508 may invoke an action based on theinterpretations provided by the above-described interpretation modules(502, 504, 506). For example, in response to the interpretation, theaction-taking module 508 may: invoke a menu; close a menu; transitionbetween workspaces (to be described below); perform a function; save asetting; present an information item, and so on.

B. Headset and User Device Options for Facilitating Interaction BetweenUsers and their Environments

FIG. 6 shows additional details regarding the headset 108, the userdevice 106, and the other user computing device 110 introduced above, inthe context of FIG. 1. The features of these devices are presentedherein in the spirit of illustration, not limitation.

Referring first to the headset 108, this device may provide a frame 602made of plastic and/or metal and/or any other material. The frame 602may be flexible, and may be secured to the user's head via tension inthe frame which pushes laterally inward against the user's head, and/orthrough some other securing mechanism. The headset 108 includestransducers (604, 606) which transmit vibrations to the bones of theuser's head; the bones of the user's head then transfer these vibrationsto the user's eardrums, where they are perceived as audio information.The use of a bone conduction-type headset prevents the headset 108 fromoccluding the user's ear canals, and thereby allows the user to safelyrespond to other sounds in the environment. However, alternatively, theheadset 108 may include conventional speakers that are placed over, ornear, the user's ears.

The headset 108 may also optionally include a set of input mechanisms608 anywhere on its frame. The user may interact with the inputmechanisms 608 with one or more fingers of his or her hand 610 (orhands). Alternatively, a separate device (not shown) may provide theinput mechanisms 608, and that separate device may communicate with theheadset 108 via wireless communication (e.g., BLUETOOTH communication)and/or wired communication. FIG. 6 shows that the input mechanisms 608include three buttons, but, more generally, the input mechanisms 608 caninclude any number of mechanisms, and these input mechanisms can beplaced on the frame in any manner. Further, the input mechanisms 608 caninclude other types of input mechanisms besides buttons, such as wheelor knob mechanisms, slider mechanisms, etc. Alternatively, or inaddition, the headset 108 may incorporate one or more touch-sensitivesurfaces. For example, different regions of the headset 108 mayincorporate different touch-sensitive mechanisms, and those regions maybe associated with different respective functions.

The headset 108 may include processing mechanisms which perform varioustasks (to be described below). The headset 108 may include a compartment612 which houses those processing mechanisms. For example, in one case,the compartment 612 lies in the back of the headset 108. However, theprocessing mechanisms can be physically located at any location (orlocations) on the headset 108. The processing mechanisms themselves mayinclude one or more processing devices of any type, memory, etc., and/ordedicated logic components, e.g., one or more application-specificintegrated circuits (ASICs), etc.

The input mechanisms 608 can initiate any operations when activated. Forexample, without limitation, the user may use the input mechanism 608 toinstruct the SI module 302 to invoke the explore mode or the orientationmode (described above). In addition, or alternatively, after hearing asummary of a certain topic (e.g., the name of a contextual IOI), a usermay use the input mechanisms 608 to instruct the SI module 302 toprovide additional information regarding the identified topic; thisinstruction may be referred to as a “more information” instruction.

In addition, or alternatively, the user can use the input mechanisms 608to instruct the SI module 302 to activate a listening mode, or to stopthe listening mode. In the listening mode, the voice interpretationmodule 504 processes the user's speech to determine whether the user hasspoken an instruction.

In addition, or alternatively, the user can use the input mechanisms 608to instruct the SI module 302 to repeat the most recent audio messagethat it has provided. In addition, or alternatively, the user can usethe input mechanisms 608 to request the SI module 302 to repeat a set ofpreviously-delivered audio messages, e.g., by starting with the mostrecent message and advancing back in time, message by message, for apredetermined number of prior messages. Such an instruction may bereferred to as a “rewind” instruction.

Alternatively, or in addition, the user may use the input mechanisms 608to turn the three-dimensional beat sound on or off, and so on. Otherimplementations can use the input mechanisms 608 to issue otherinstructions, and/or to omit one or more of the instructions set forthabove.

The above functions, or some subset thereof, can be mapped to any numberof respective buttons and/or other input mechanisms in any manner.Further, in some implementations, the system 102 may include acustomization module that allows a user to define the mapping betweeninput mechanisms and operations which the mechanisms invoke. Further, asdescribed above, the same button can also perform two or more functionsdepending on the manner in which the user interacts with it. In oneimplementation, the SI module 302 may announce the function performed bya button when the user touches it, but does not press it.

According to another feature, the user may deactivate a function that iscurrently in active state by pressing the same button that was used toactivate it. For example, the user can stop the SI module 302 fromannouncing information by pressing the same button that was used to askthe SI module 302 to deliver this information, etc. Alternatively, or inaddition, the headset 108 can incorporate a dedicated button which stopswhatever function is currently being executed.

The user device 106 may correspond to any of the types of portabledevices described above, such as a smartphone or a tablet-type computingdevice. As described above, a user may interact with the user device 106using a single hand 614 (and/or optionally with two hands). The otheruser computing device 110 may correspond to any type of traditionallystationary computing device, such as a workstation computing device, agame console, a set-top box device, etc.

One or more communication paths 616 may couple the user device 106 withthe headset 108. Such a communication path, for example, may correspondto a BLUETOOTH communication path, a hardwired communication path (e.g.,a USB communication path), etc. One or more communication paths 618 maycouple the other user computing device 110 to the user device 106 and/orthe headset 108. As described above, one reason that a user may wish toestablish a connection between the other user computing device 110 andthe user device 106 and/or the headset 108 is to upload information tothese devices. The communication paths 618 can be implemented in anymanner, e.g., via any type of wireless and/or wired communication pathdescribed above.

Other implementations of the system 102 can use different types ofheadsets, e.g., compared to the particular type of headset 108 shown inFIG. 6. For instance, in another implementation, a headset canincorporate any of the above-identified features (including the inputmechanisms 608) together with a head-mounted display (HMD) device of anytype, e.g., as physically implemented as eyewear (e.g., goggles), ahelmet, etc. The system 102 may display any type of information usingthe head-mounted display device. For example, the system 102 may displayany of the menus and other information described in the next section(Section C) via the head-mounted display device, or some subset thereof.In another case, the system 102 may display computer-generatedinformation that is mixed with information associated with the realworld with which the user interacts, to thereby provide an augmentedreality experience. For example, the system 102 can display descriptivetags in positional proximity to associated objects and events within thefield of vision of the user. In another case, the system 102 can displaydirectional prompts which assist the user in moving in a recommendeddirection, and so on. Moreover, the system 102 can modify the type ofinformation that is displayed to accommodate any visual impairments thatmay affect a particular user, e.g., by displaying simplified andenlarged information for users with impairments in sight.

The system 102 can achieve an augmented reality experience using anytechnology, e.g., through the use of an optical mixer which displayscomputer-generated information “over” the user's direct visualperception of the actual environment (e.g., using partially reflectivemirrors or the like). In another implementation, the system 102 can usea video camera to capture the actual environment, together with anoptical mixer which mixes video information from the video camera withcomputer-generated information. In either case, the system 102 candetermine the user's presumed field of view using one or more deviceswhich detect the location of the user within the environment, one ormore devices which detect the position and orientation of the user'shead, and/or one or more devices which detect the direction of theuser's gaze, etc.

In another case, another type of wearable device, besides a headset, orin addition to a headset, can perform any of the functions set forthabove. For example, the wearable device may correspond to awrist-mounted device, an item of apparel, etc.

To simplify the explanation, the following description will assume thatthe user interacts with the headset 108 of the basic type shown in FIG.6, although, to repeat, the headset 108 can incorporate any number ofsupplemental features mentioned above (such as a head-mounted display).Further, another type of wearable device can be used instead of aheadset, or in addition to a headset.

FIG. 7 shows one manner of implementing the system 102 of FIG. 1. Inthis implementation, the system 102 makes use of both a user computingdevice 702 (a “user device” for brevity) and a headset 704. Morespecifically, according to one allocation of functions, the user device702 implements most of the functionality of the system 102. That is, theuser device 702 includes the SI module 302 and a collection ofinput/output mechanisms 706, including any orientation, motion, and/orposition determination mechanisms 708, any touch-sensitive inputmechanisms 710, any haptic output mechanisms 712, any display outputmechanisms 714, and so on. Examples of these mechanisms were providedabove in connection with the description of FIG. 3. The user device 702also includes a power source 716 (e.g., a battery) and one or morecommunication mechanisms 718 for communicating with the headset 704.

On the other hand, the headset 704 includes just one or more audiooutput mechanisms 720 (such as a bone conducting audio mechanism), apower source 722 (such as a battery), and one or more communicationmechanisms 724 for communicating with the user device 702. In addition,the headset 704 may include any of the type of input mechanisms 726described above with reference to FIG. 6 (e.g., corresponding to theinput mechanisms 608 of FIG. 6). The communication mechanisms 724 cantransmit instructions, invoked by the input mechanisms 726, to the userdevice 702. The user device 702, in turn, can send audio information tothe headset 704 for presentation by the audio output mechanism(s) 720.

FIG. 8 shows another way of implementing system 102. Here, the system102 includes a headset 802 having more processing capability compared tothe headset 704 of FIG. 7. Indeed, in one implementation, the headset802 may perform all space-interaction-related aspects of the system 102,without the use of any separate user device 106. In anotherimplementation, the headset 802 may still interact with a separate userdevice 106 (not shown in FIG. 8). That separate user device 106 caninclude all of the components of the user device 702 of FIG. 7, or anysubset thereof.

More specifically, the headset 802 in the case of FIG. 8 may include aheadset-side SI module 804 that performs any subset of the functionsdescribed above with reference to FIG. 4. In addition, the headset 802may include any orientation, motion, and/or position determinationmechanisms 806. These mechanisms are generally referred to below ashead-sensing mechanisms because they determine the physical posture ormovement of the user's head, which may, in turn, reflect the directionof the user's focus of attention. In addition, the headset 802 mayinclude any of the type of input mechanisms 808 described above withreference to FIG. 6. In addition, the headset 802 may include a powersource 810 and one or more communication mechanisms 812 for interactingwith the user device 106 (if the system 102 makes use of such a userdevice 106 in this implementation, which it need not). In addition, theheadset 802 includes any type(s) of audio output mechanism(s) 814described above.

In one mode of operation, the system 102 of FIG. 8 can make use of thehead-sensing mechanisms provided on the headset 802 when they areprovided and working properly. The head-sensing mechanisms provided bythe headset 802 may be preferable to the counterpart sensing mechanismsprovided by the user device 106 because the head-sensing mechanisms maymore accurately reflect the orientation, movement, and/or position ofthe user. Further, the use of the head-sensing mechanisms eliminates theneed for the user to interact with the user device 106 to register hisor her orientation, movement, and/or position.

In some cases, the headset 108 may optionally forward the informationgenerated by the head-sensing mechanisms to the user device 106. Theuser device 106 may use this information to perform processing and thenforward the results of its processing back to the headset 802. Theheadset 802 may then transmit sounds to the user's ears via a boneconducting technique that convey the results. In another case, theheadset-side SI module 804 can perform this processing without the needfor forwarding the information provided by the head-sensing mechanismsto the user device 106. In still another case, the headset-side SImodule 804 can natively perform some operations, and rely on the userdevice 106 to perform other operations. For example, the headset-side SImodule 804 can rely on the user device 106 to perform computationallyintense operations (such as the calculation of three-dimensional sounds,etc.) because those operations can be more efficiently performed on theuser device 106 compared to the headset 108.

In another case, the system 102 may make use of the orientationdetermination mechanism(s), motion determination mechanism(s), and/orposition determination mechanism(s) provided by the user device 106 whenthe counterpart components are not working properly on the headset 802.

In still another case, the system 102 may leverage the sensor readingsprovided by both the headset 802 and the user device 106, e.g., by usingsensor readings of one device to identify obvious errors in the sensorreadings of the other device, and/or to form an average of the twodifferent versions of the sensor readings, and so on.

In either of the implementations of FIGS. 7 and 8, at least some of thefunctions performed by the headsets (e.g., 704, 802) and/or the userdevices (e.g., 702, 106), can be alternatively, or in addition,performed by the remote processing resources 112 of FIG. 1 (e.g.,corresponding to cloud computing resources).

FIG. 9 shows a scenario in which a user 902 interacts with a user device904 within a vehicle. Although not specifically illustrated in FIG. 9,the user 902 may also make use of a bone conducting headset that relaysaudio information to the user without occluding the user's ears. Moregenerally, FIG. 9 is an example of the more general point that thefunctionality described above can be applied to additional use scenariosthan the cases described above.

In the case of FIG. 9, the user 902 has mounted the user device 904,using a mount 906, onto the dashboard-region of a vehicle. The user 902may be the driver of the vehicle (as shown in FIG. 9) or a passenger. Apower cord 908 may deliver power to the user device 904 from a poweroutlet provided by the vehicle.

FIG. 10 shows a process 1004 which describes one manner of operation ofthe equipment shown in FIG. 6. In block 1004, the system 102 receives aninstruction as a result of the user's actuation of at least one headsetinput mechanism (e.g., based on the user's actuation of one of theheadset input mechanisms 608 shown in FIG. 6). In block 1006, the system102 performs a function based on the instruction provided in block 1004,to provide an output result. Block 1006 may be performed by the headset108 and/or the user device 106. In block 1008, the headset 108 appliesthe output result to deliver audio information, which assists the userin navigating over a route, within a space, or, more generally,interacting with the space.

In summary, the above features contribute to the above goal of allowingthe user to safely and efficiently move through his or her environment.For instance, the features provide a convenient way by which the usermay activate various operational modes (e.g., by interacting with theuser input mechanisms 608 of the headset 108) without unduly distractingthe user as the user moves through the environment, e.g., withoutrequiring the user to access and interact with a separate user device.

C. Illustrative User Interface Functionality for FacilitatingInteraction Between Users and their Environments

This section provides illustrative details regarding one manner ofoperation of the application interface module 402 of FIGS. 4 and 5. Torepeat, the application interface module 402 allows the user to interactwith the various components of the SI module 302. To facilitate repeatedreference to the application interface module 402, this section willmake reference to this component in abbreviated form as the AIM 402. TheAIM 402 may refer to a discrete component, or to an aggregation offunctions performed by two or more components in an actualimplementation of the system 102.

The user interface experience, in turn, has different components oraspects. Subsection C.1, below, provides details regarding oneillustrative manner in which the AIM 402 allows a user to interact withthe SI module 302 via a visual user interface presentation, e.g., asprovided by the user device 106. Subsection C.2 provides detailsregarding one illustrative manner in which the AIM 402 may providevarious sounds and haptic feedback cues to the user. Subsection C.3provides details regarding one illustrative manner in which the AIM 402allows the user to interact with the SI module 302 via spokeninstructions.

Generally note that the following explanation describes the AIM 402 inthe context of a journey taken by a user through space, e.g., as in theexample of FIG. 2. But the AIM 402 provides a similar service in otherscenarios in which a user interacts with a space, including the case inwhich the user wanders through the space without a prepared route forthe purpose of exploring the space in an ad hoc manner. As a furthernote, the user interface features described below are also generalpurpose in nature, and thus can be applied in other contexts that arenot necessarily related to a user's interaction with a space.

As another prefatory note, the description sets forth many gesturesinterpreted by the AIM 402 by describing gesture-related actionsperformed by a user, together with corresponding operations taken by theAIM 402 in response to the gesture-related actions. As more fullydescribed above with reference to FIG. 5, the AIM 402 performs this taskin each case by: (a) detecting input information which describes agesture performed by the user, for example, when the user makes atelltale flick gesture, etc.; (b) comparing the input information withstored patterns associated with known gestures, to determine theparticular gesture that the user has invoked; and (c) executing theoperations associated with the detected gesture. Explicit recitation ofthese individual operations is omitted in many cases below in theinterest of brevity.

C.1. Visual Experience: Interacting with Workspaces and Menus

In one approach, the AIM 402 organizes a master workspace into acollection of smaller workspaces. The workspaces have positionalrelationships with respect to each other. This structure is beneficialbecause it allows the user to develop a mental picture of theorganization of the application, much like he or she would becomefamiliar with the zones of a physical space through repeated encounterswith its zones. This feature, in turn, allows the user to efficientlyaccess the information and/or functionality being sought.

For instance, FIG. 11 shows an example in which the AIM 402 organizes amaster workspace into five smaller workspaces. A home workspace 1102serves as the central focus in the user's interaction while conducting ajourney. For example, as will be described below, the home workspace1102 presents information regarding the current context of the user, ateach particular point in the journey or any other interaction with aspace. If the user has not yet started the journey, the home workspace1002 may present a default hub page.

A main workspace 1104 lies to the left of the home workspace 1102, and a“nearby me” workspace 1106 lies to the right of the home workspace 1102.An information workspace 1108 lies to the top of the home workspace1102, while a settings workspace 1110 lies to the bottom of the homeworkspace 1102. The respective roles of these workspaces will bedescribed in greater detail below.

The position of each workspace described above is set forth in thespirit of illustration, not limitation. Other implementations canprovide other placements of the workspaces. Further, otherimplementations can vary the number of workspaces that are provided bythe AIM 402. For example, another implementation can provide anotherfour workspaces by providing workspaces that are diagonally positionedrelative to the home workspace 1102.

FIG. 12 shows one manner by which a user may navigate from the homeworkspace 1102 to the main workspace 1104. In this non-limiting case,the user executes a flick-right gesture on the home workspace 1102 tonavigate to the main workspace 1104. Although not shown, the user mayexecute a flick-left gesture (or another flick-right gesture) to movefrom the main workspace 1104 back to the home workspace 1102.

In a similar manner, the user may execute a flick-left gesture on thehome workspace 1102 to navigate to the “nearby me” workspace 1106. Theuser may execute a flick-down gesture on the home workspace 1102 tonavigate to the information workspace 1108, and a flick-up gesture onthe home workspace 1102 to navigate to the settings workspace 1110. Theuser can perform any of these gestures, for example, using the thumb ofthe hand that holds the user device, e.g., by placing the thumb on thetouch-sensitive surface of the user device 106 and flicking it in adesired direction. In a flick movement, the user makes contact with thetouch-sensitive surface with one or more fingers, moves the finger(s)for a distance across the surface, and then removes the finger(s) fromthe surface, all in relatively rapid succession (as if flicking aphysical object across a flat surface).

In one implementation, in order to navigate to any peripheral zone, theuser is expected to first navigate to the home workspace 1102. However,in another implementation, the AIM 402 may allow the user to navigatefrom one peripheral region to another without first moving back to thehome work space 1102.

More generally, in all examples in this section, a dashed-line circle onthe surface of a display represents the point at which the user contactsthe surface with a digit. In some cases, the surface of the display willshow two dashed-line circles of different sizes. The larger of the twocircles represents the location at which the user applied his or herdigit, while the smaller of the two circles represents the location atwhich the user removed his or her digit. In other cases (not shown), auser may perform at least one gesture that involves simultaneouslytouching the touch-sensitive surface at two or more locations (e.g., toexecute a pinch-type gesture). In other cases (not shown), a user mayperform at least one gesture by hovering above the surface of the devicewithout actually touching it. In other cases (not shown), a user mayperform a free-space gesture, which may be detected by a video cameraand/or a depth camera, etc.

As a final introductory note, this section describes particulargestures, menu structures, and menu items, all in the spirit andillustrative, not limitation. Other implementations can vary any aspectsof these user interface features. For example, in another implementation(not illustrated), the AIM 402 could allow the user to transition amongworkspaces using a tap-type gesture, or by drawing a particular shape onthe surface of the display, etc.

FIGS. 13-16 illustrate the types of information that may be presented inthe home workspace 1102. As mentioned above, the home workspace 1102serves as the primary focus of the user's attention as they navigateover a route or otherwise interact with the space. In a default state,as shown in FIG. 13, the home workspace 1102 may present a hub page. Theparticular nature of the menu structure, and the menu items within thatstructure, will be described below.

In the course of the user's journey, the home workspace 1102 may presentinformation regarding the user's current context (although otherinformation can also be presented in the home workspace 1102 during thejourney, upon request by the user). For example, in the case of FIG. 14,assume that the user is currently riding on a train. The home workspace1102 may present a menu that focuses on the user's experience in thetrain. In the case of FIG. 15, assume that the user is now walking downa particular street. As a default, the home workspace 1102 will nowpresent information regarding the user's experience on the street. Forexample, the home workspace 1102 can present a map that shows the user'scurrent location 1502 on the street. Alternatively, the user mayinteract with the AIM 402 to produce the type of menu shown in FIG. 14,but where that menu will now contain menu items that pertain to theuser's experience while walking on the street. By virtue of the abovebehavior, the AIM 402 surfaces the information that is most relevant tothe user in his or her current context in the environment, and presentsthat information within the home workspace 1102; the user can thusreadily find and consume the most relevant information without having to“hunt” for it, e.g., without having to navigate through a cumbersomemenu structure. Such a feature also reduces the distractions placed onthe user, and thus contributes to the safety of the system 102.

The user may nevertheless interact with the AIM 402 to activateinformation, for presentation in the home workspace 1102, that does notpertain to the user's immediate surroundings. For example, the user mayactivate a tab menu, e.g., by activating a tab menu option, to bedescribed below. In response, the AIM 402 will present the tab menushown in FIG. 16. The tab menu shows a collection of open tabs,corresponding to previously opened menus. Some of these tabbed menus maycorrespond to information regarding prior journey segments that the userhas already completed. The user may activate any such tab-related menuitem. In response, the AIM 402 may represent information regarding aprior journey step in the home workspace 1102, e.g., in the form of amenu or some other format. The menu structure may also allow the user toexamine future journey steps, upon request by the user, that have notyet been encountered on the journey.

Further, even after having embarked on a journey, the user may instructthe AIM 402 to return to the hub menu shown in FIG. 13, e.g., so thatthe user may access the information and/or functionality specified inthat default hub menu.

As a further clarification, in the example above, the current contextpertains to a physical location or segment in the user's overalljourney. In other cases, the home workspace 1102 can present contextinformation regarding the user's exploration of a virtual space. Forexample, the user can navigate within a hierarchy of products offered bya store. In that scenario, the home workspace 1102 may present a menuthat pertains to a group of products, an individual product, etc.

Generally speaking, the above-described types of menus shown in FIGS.13-16 correspond to primary menus. The AIM 402 displays these types ofmenus in the home workspace 1102. In contrast, the AIM 402 presentssecondary menus in the other workspaces that lie on the periphery of thehome workspace 1102. FIGS. 17-20 show illustrative secondary menus thatmay be presented in these peripheral workspaces.

For example, FIG. 17 shows a main menu for presentation in the mainworkspace 1104. The main menu identifies actions that are commonly usedwhen interacting with different aspects of the SI module 302. FIG. 18shows a settings menu for presentation in the settings workspace 1110.The settings menu allows the user to change various parameters whichaffect the operation of the SI module 302. FIG. 19 shows an informationmenu for presentation in the information workspace 1108. The informationmenu presents convenient access to system information, such as remainingbattery life, signal strength. The information menu also provides aconvenient portal to notifications and other useful warning informationand journey information, such as real-time updates from public transportservices.

FIG. 20 provides a “nearby me” menu for presentation in the “nearby me”workspace 1106. The “nearby me” menu presents information regardingcontextual items of interest (IOIs) that lie within close distance tothe user at the present time (or are otherwise associated with theuser's current context), as identified by the RID module 426 of FIG. 4.The user may use the settings menu to specify a maximum distance whichdetermines what constitutes a “nearby” contextual IOI. Each contextualIOI is identified in the “nearby me” menu by its name, its category(such as the “Food and Drink” category), and its distance from the user.A user may activate an item in this menu, causing the SI module 302 toprovide audio information regarding this contextual IOI. That audioinformation may be formulated as three-dimensional sound information,such that, upon delivery, it appears to originate in the same directionat which the entity associated with the contextual IOI can be found inphysical space.

FIG. 21 shows an example of a transient menu that may be presented inthe home workspace 1102. For example, a user may activate this transientmenu by first navigating to an information menu which is shown in FIG.19, and which is presented in the information workspace 1108. The usermay then activate the “notifications” menu item in the menu. Inresponse, the AIM 402 may present the transient menu shown in FIG. 21.

FIG. 22, by contrast, shows an example of an overlay menu. In thisparticular example, the user may activate the overlay menu by firstadvancing to the settings menu shown in FIG. 18, as presented in thesettings workspace 1110. The user may then activate the “contextualawareness” menu item in that menu. The AIM 402 responds to the user'saction by presenting the overlay menu shown in FIG. 22. Unlike thetransient menu shown in FIG. 21, the overlay menu shown in FIG. 22 ispresented “over” the settings menu shown in FIG. 18 in the settingsworkspace 1110, rather than in the home workspace 1102. Overlay menusand transient menus may also exhibit different behaviors in response tothe execution of a “back” command, as will be described below.

In the particular case of FIG. 22, the overlay menu allows the user tochange the level of contextual awareness provided by the SI module 302.For example, through this menu, the user can set a parameter value whichdetermines the quantity of information that is forwarded to the user. Inone case, the user may opt to receive all information that ispotentially relevant to his or her current context. In another case, theuser may opt to receive only warnings and other information items thatare deemed of high importance.

FIG. 23 demonstrates one manner by which the user may instruct the SImodule 302 to present information regarding his or her current context,at any juncture in the user's journey. According to one non-limitingcase, the user may use a digit (e.g., a thumb) to make a single-tapgesture on the touch-sensitive surface of the user device 106, at anylocation on the surface. In response to detecting this gesture, the SImodule 302 can present, as a spoken message, high-level informationregarding the user's current context. For example, the SI module 302 canannounce the title of the context menu that would be appropriate forthis context. For instance, in the example of FIG. 14, the SI module 302could announce, “Carriage E: London Paddington.”

Alternatively, the user may make a double tab gesture. In response todetecting this gesture, the SI module 302 can present more detailedinformation regarding the user's current context, again as a spokenmessage. For example, upon a double tap, the SI module 302 can announcethe menu items in the menu shown in FIG. 14.

Next assume that the user performs a triple tap gesture. In response todetecting this gesture, the SI module 302 can announce instructions thatinform the user how to interact with the type of menu shown in FIG. 14,if that menu, in fact, pertains to the user's current context. Forexample, the instructions can inform the user how to activate the menu,how to navigate within the menu, how to select menu items within themenu, and so on.

Advancing to FIG. 24, this figure shows how a user may activate a menu,and then how the user may subsequently interact with the menu. In onenon-limiting case, the user may activate the menu shown in FIG. 24 bymaking a tap-and-hold gesture anywhere on the surface of thetouch-sensitive display surface of the user device 106. For example, asshown in FIG. 24, a dashed-lined circle 2402 indicates the location atwhich the user has tapped and held his or her thumb on the surface ofthe touch-sensitive surface.

In response to detecting this gesture, the AIM 402 presents the menu2404. The menu is centered at the point (corresponding to a displaylocation) on the surface at which the user touched the surface(corresponding to a touch location). The user may find this featureuseful because it eliminates the need to hunt for a particular elementin a user interface presentation for the purpose of activating the menu2404. Rather, the user may execute the tap-and-hold gesture anywhere onthe surface. This feature, in turn, reduces the distractions placed onthe user in using the system 102. But not all locations will result inthe successful activation of the menu 2404. For example, if the usertaps too close to the top or bottom of the surface, the AIM 402 mayoptionally present an error message to the user (e.g., as a spokenmessage), asking the user to repeat his or her tap-and-hold gesturecloser to the middle of the surface.

The menus illustrated thus far have a particular uniform menu structure.The menu 2404 shown in FIG. 24 has the same structure, which will now bedescribed below, with the caveat that this structure is set forth in thespirit of illustration, not limitation. Other implementations can adoptother menu structures and behaviors. For example, this subsection willclose with an example of an implementation which adopts a different menustructure compared to the menu 2404 of FIG. 24, and which exhibits adifferent menu behavior. Further, the menus described herein presenttheir menu items in the form of linear lists; but other menus can beused (e.g., radial or pie menus, etc.), which present their menu itemsin other ways.

The menu structure shown in FIG. 24 has two groupings of menu items,separated by a marker item 2406. More specifically, a first grouping2408 of menu items presents menu items that are particularly pertinentto the user's current context or task at hand. A second grouping 2410 ofmenu items presents menu items that are relevant to multiple types ofmenus that may be presented in different workspaces, and therefore maybe referred to as global or general menu items. For example, the “stoplistening” command, which is a menu item in the second grouping 2410, isrelevant across different menus and workspaces, whereas the menu item“item A,” which is a menu item in the first grouping 2408, may be chosenbecause it is particularly relevant to whatever task that the user isattempting to accomplish by interacting with the menu 2404.

The second grouping 2410, however, may omit certain global choices thatare not relevant for a particular menu. For example, if the user isalready viewing a “tabs” menu, the second grouping associated with thatmenu may omit a menu option that allows the user to access the tabs menu(because the user is already viewing that menu).

Different menu items (in either the first grouping 2408 or the secondgrouping 2410) invoke different operations when selected. For example, afirst type of menu item may invoke an action when selected. For example,the menu item “start listening,” when invoked, instructs the SI module302 to start listening for the user's voice commands. A second type ofmenu item may present information when invoked, such as battery statusinformation, etc. A third type of menu item may present an opportunityfor the user to change a property value, when that menu item is invoked.For example, in some cases, the user's activation of this type of menuitem may activate an overlay menu. The user may then interact with theoverlay menu to change a property value under consideration. In anothercase, the user's activation of this type of menu item may directlychange the property value, e.g., by toggling a setting from an “on”status to an “off” status, or vice versa. Still other types of menuitems and associated invocation behaviors are possible.

The marker item 2406 displays the message “release to dismiss.” Thismessage informs the user that he or she can release their finger fromthe surface of the touch-sensitive surface without selecting any menuitem, assuming, that is, that the user releases his or her finger whileit was positioned on the marker item 2406, and not some other menu item.In response to such a gesture, the AIM 402 can display the menu 2412 inits original deactivated state. The marker item 2414 in that menu 2412bears the message “tap and hold,” which invites the user execute atap-and-hold gesture to reactivate the menu 2404 in its active state.

Advancing to FIG. 25, assume that, instead of deactivating the menu2404, the user decides to scroll up through items in the first grouping2408. The user may perform this action by moving his or her finger inthe upward direction, while maintaining his or her finger on thetouch-sensitive surface. In response, the AIM 402 can produce the menustate 2502 shown in FIG. 25. In particular, the AIM 402 has responded tothe user's gesture may moving the menu items in the downward direction,which causes a first menu item 2504 to become highlighted, instead ofthe marker item 2406. To select this menu item 2504, the user mayrelease his or her finger from the menu item 2504. Alternatively, theuser may deactivate the menu by moving back to the marker item 2406 andthen releasing his or her finger while positioned at that item. Or theuser may scroll down in the opposite direction to the second grouping2410 of menu items. The user can release his or finger while it ispositioned on one of the menu items in the second grouping 2410 toselect that menu item. A user may find the above user interface behaviorbeneficial because he or she can interact with the user device 106 withone hand in a seamless and fluid manner, with minimal distractionsplaced on the user as he or she moves through the environment. Forinstance, in this implementation, the user is not required to hunt forand select a series of command buttons or menu items; doing so wouldrequire the user to give special attention to the user device 106.

In the specific example of FIG. 25, assume that the user scrolls up tothe top of the first grouping 2408 of items to select a “more items”menu item 2506, and then releases his or her finger on this menu item2506. In response, the AIM 402 presents the menu 2602 shown in FIG. 26.That menu provides another first grouping 2604 of menu items thatrepresents a continuation of the first grouping 2408 shown in theprevious menu; that is, while the first grouping 2408 in FIG. 25presents menu items 1, 2, and 3, the first grouping 2604 of FIG. 26presents menu items 4, 5, and 6. More generally, the menu structure mayrepresent a complete list of menu items as a series of linked smallerlists. Two linked lists are shown in the example of FIGS. 25 and 26, butthe complete list can be formed through any number of linked lists.

The user may return to the menu state 2502 shown in FIG. 25 by scrollingup to the “previous page” menu item 2606, and then releasing his orfinger from this item. Alternatively, to execute this back instruction,as shown in FIG. 26, the user may move his or her finger to theperiphery of the touch-sensitive surface (e.g., as indicated by thedashed-line circle 2608), and then flick inward towards to the center ofthe surface. The user may execute the same operation by flicking inwardfrom the opposite edge of the surface.

Although not shown in FIG. 26, the menu 2606 may also give the user theoption (via an appropriate menu item) to return to the first page in aseries of cascaded menu lists, e.g., rather than successively movingback through the linked lists by issuing a series of back instructions.Further, although also not shown in the drawings, each grouping mayinclude an end-of-list marker item, which designates the end of agrouping. The user can deactivate the menu by moving to this item andremoving his or her finger while it is placed on that item.

More generally, the user may execute the type of back gesture shown inFIG. 26 on different types of menus, which may produce different typesof actions. For example, a back action on a root (primary) menupresented in the home workspace 1102 may cause the user to exit theapplication associated with the SI module 302, upon confirmation by theuser that this is what he or she intends to do. A back action on atransient menu that is presented in the home workspace 1102 may causethe AIM 402 to present whatever context menu was last presented in thehome workspace 1102. A back action on an overlay menu may sometimesresult in the presentation of an “underlying” secondary menu in asecondary workspace. For example, a back action on a particularsettings-related overlay menu (e.g., as shown in FIG. 22) may result inthe display of an underlying settings menu (e.g., as shown in FIG. 18)in the settings workspace 1110.

FIG. 27 shows an alternative organization of workspaces, compared to theimplementation of FIG. 11, corresponding to a second implementation ofmenu functionality provided by the AIM 402. By way of overview, theworkspace structure of FIG. 27 includes five workspaces (2702, 2704,2706, 2708, 2710) that serve the same basic functions as those shown inFIG. 11. For example, the home workspace 2702 continues to serves as thecentral focus of the user's attention throughout a journey. The homeworkspace 2702 also continues to display menus and other informationassociated with the current context encountered in the user's journey(or other type of interaction with a space). But the home workspace 2702no longer presents a hub menu as a default. Rather, the home workspaceis devoted to showing tab-related information as a default.

More specifically, the home workspace 2702 shows the default tabsinformation shown in FIG. 28 when there are no active tabs. The defaulttabs information may provide guidance on how the user may get started tocreate tabs. In contrast, the home workspace 2702 may display the tabsmenu shown in FIG. 29 when there are active tabs. By default, the tabsmenu item having the first position in the list (i.e., tab No. 1)corresponds to a current context.

FIG. 29 also shows that the second implementation offers a differentmenu structure compared to the menu structure described thus far. Forexample, the second implementation no longer organizes the menu into twogroups of menu items, separated by a menu marker. Rather, the secondimplementation presents a single list of menu items that are determinedto be relevant to the user's current focus of interest. If the userwishes to access the type of global menu items that were previouslypresented in the second grouping, the user may navigate to the mainworkspace 2704 to access those items.

FIG. 30 presents additional information regarding one possibleorganization of menu items in a list of menu items. Here, the menu itemsrepresent search result items that are produced in response toconducting a search for shops that are nearby the user. The user mayactivate any menu, such as the menu shown in FIG. 30, by performing thetap-and-hold gesture described above with respect to the firstimplementation (and as illustrated in FIGS. 23 and 24).

Upon activation of the menu, in the second implementation, the AIM 402presents the first entry in the list near the middle of the userdevice's display surface. The user may then move up through the list bymaking one or more panning gestures in an upward direction, or move downthrough the list by making one or more panning gestures in the downwarddirection. When the user reaches the end of the list in eitherdirection, the list repeats. That is, when the user advances past thelast item in the list, the user will encounter, as a next entry, thefirst item. When the user advances past the first item in the list, theuser will encounter the last item. Due to this menu presentationstrategy, the second implementation may dispense with the use ofmultiple-page lists, as used in the first implementation. That is, theuser navigates through a single master list using one or more panninggestures. A user makes a panning gesture by placing one or more fingersin contact with the touch-sensitive surface, and dragging thosefinger(s) in a desired direction of panning; the movement here is slowercompared to a flick gesture.

FIG. 31 provides additional information regarding the above-describedbehavior. The user here makes a panning gesture by dragging thedisplayed menu in the downward direction. But unlike the case of thefirst implementation, the user need not keep his or her finger on thesurface when navigating through the list. For instance, the user maymake plural panning gestures; between each panning gesture, the user mayremove his or her finger from the surface. The second implementationwill not interpret the removal of the user's finger from the surface asan instruction to select whatever menu item happens to be highlighted atthat time. The second implementation may deactivate a menu after aprescribed amount of time of inactivity by the user, or when the usermakes a back gesture.

As shown in FIG. 32, a user may select a highlighted item in the list bymaking a short touch-and-drag gesture towards the periphery of thesurface. The AIM 402 will provide audible and/or haptic feedback when ithas interpreted the user's gesture as a request to select thehighlighted item. Then, the AIM 402 will select the item when the userremoves his or her finger from the surface of the user device 106.Alternatively, instead of removing his or her finger, the user may draghis or her finger back in the opposite direction to cancel the selectionoperation, which the AIM 402 will again confirm (e.g., by providing anappropriate cue or cues).

FIG. 33 shows a back gesture performed at the periphery of a menu, andthe response to this gesture. More specifically, the user may nowperform a back gesture by pulling left or right from the respective edgefor a short distance until the AIM 402 indicates that it has understoodthe user's gesture (e.g., by providing an appropriate cue or cues). Atthat time, the user can release his or her finger to perform a backaction. Or the user may move his or finger in the opposite directionuntil the AIM 402 interprets the user's action as a request to revokethe back action. In the example of FIG. 33, the back action causes theAIM 402 to transition from a search results menu to a tabs menu.

FIG. 34 shows a gesture that a user may perform by drawing a circleshape 3402 on a menu, e.g., by tracing out a circle on a search resultsmenu (for example) that is presented in the home workspace 2702. Inresponse to this gesture, the AIM 402 will return to the tabs menu.Although not shown, the user may subsequently perform the same circlegesture on the tabs menu. In response, the AIM 402 will return to thesearch results menu (or whatever page from which the useroriginated—that is, whatever page on which the user drew the originalcircle).

FIG. 35 shows a gesture that the user may perform by drawing ahalf-circle 3502 on a menu, e.g., by tracing out a half-circle on asearch results menu (for example) that is presented in the homeworkspace 2702. In response, the AIM 402 will return focus to whateverwas being presented in the current context. In the example of FIG. 35,the AIM 402 returns to information associated with a current step in thejourney (as shown on the right side of FIG. 35). If the user is notcurrently undertaking a journey, the current context may correspond towhatever menu was opened last. The user can reverse this transition bydrawing a half-circle on the page shown on the right side of FIG. 35.

FIGS. 36 and 37 show different gestures that can be used to increase anddecrease, respectively, a level of verbosity provided by the system ofFIG. 1, and a level of contextual information provided by the system.For example, as indicated in FIG. 36, the user may draw a right-arrowshape 3602 on any menu (here, a tabs menu) to increase the verbositylevel of the SI module 302. As indicated in FIG. 37, the user may drawthe left-arrow shape 3702 on any menu to decrease the level of verbosityof the SI module 302. The level of verbosity refers to the brevity atwhich the SI module 302 delivers its audio information, e.g., rangingfrom terse to detailed.

As also indicated in FIG. 37, the user may draw an up-arrow shape 3704or a down-arrow shape 3706 on any menu to increase or decrease,respectively, a level of information that is provided to the user. Forexample, the user may scale back on the quantity of contextualinformation by refraining from sending the user information regardingone or more of the least critical categories of information items (suchas by omitting contextual information, but sending warning informationand journey information).

The second implementation may also execute other new gestures (comparedto the first implementation), although not specifically illustrated inthe figures. For example, the user can perform a tap-and-hold gesture ona first menu to transition from that menu to an actions menu (where theactions pertain to actions that are relevant to the context associatedwith the first menu). The user may perform the same gesture on theactions menu to return to the original (first) menu. The tap-and-holdgesture in this case involves a longer hold action than the tap-and-holdgesture that is used to generally activate any menu.

As another new gesture, the user may perform a vertical flick gesturewhile on the menu items of a menu. The AIM 402 will interpret thisaction as a request to quickly advance through the menu items, in eitheran up or down direction, depending on the direction of the flick. Asanother new gesture, the user may perform a horizontal flick gesture ona menu item in a menu to advance to the next menu item in the list. Theuser can make multiple such horizontal flick features to successivelyadvance through the list, one menu item at a time.

In either of the first or second implementations (corresponding to FIGS.11 and 27), the user may execute a gesture to place the user device 106in a pocket mode. When the user device 106 detects that gesture, it willignore any subsequent gestures that the user may make, with theexception of a gesture that has the effect of canceling the pocket mode.As the name suggestions, the pocket mode is useful in those cases inwhich the user wishes to stow the user device 106 in a pocket or someother compartment (such as a purse, bag, etc.). When active, the pocketmode prevents accidental touch contact and/or movements by the user asbeing (incorrectly) interpreted as meaningful gestures. The gesturesused to invoke and revoke the mode can correspond, for instance, totelltale swipe gestures, etc.

C.2. Sounds and Haptic Cues

The AIM 402 can present various sounds by leveraging the use of thesound generation module 414. The AIM 402 can also present various hapticfeedback experiences (e.g., vibration cues) using the haptic cuegeneration module 418. The AIM 402 can produce such sounds andvibrational cues in response to different types of triggering events.For example, the AIM 402 can present these sounds and haptic cues inresponse to certain changes in state, in response to certain interfaceactions taken by the user (such as navigation among menu items), inresponse to certain events in a journey, and so on.

The following listing describes representative sounds (and associatedhaptic cues), and the illustrative circumstances in which they areinvoked.

Loading/Busy. The AIM 402 can repeatedly play a Loading/Busy sound whilethe SI module 302 is in a state in which the user is unable to interactwith it. The sound reassures the user that the SI module 302 is workingbut it is presently busy performing an action. For example, this soundmay resemble a ping pong ball bouncing up and down; the frequency of thebounce may increase as the processing nears completion, and may end witha flourish-type sound.

Transition-To-Zone-Menu. This sound indicates that a secondary menu,presented in one of the dedicated secondary workspace zones, has movedinto focus and is now being presented instead of the current contextmenu in the home workspace. This sound may also convey the direction inwhich the user has performed the corresponding gesture. For example, thesound may resemble a swoosh, like a gust of wind in a respectivedirection. Further, in one case, the AIM 402 may express this sound as athree-dimensional sound.

Transition-From-Zone-Menu. This sound indicates that the current contextmenu has moved back into focus and is now being presented instead of asecondary menu in one of the dedicated workspace zones. This sound mayalso convey the direction in which the user has performed thecorresponding gesture. For example, the sound may resemble a swoosh thatis the counterpart of the Transition-To-Zone-Menu swoosh, but moving inthe opposite direction than the Transition-To-Zone-Menu sound. Again,the AIM 402 may optionally express this sound as a three-dimensionalsound.

Menu-Activation. This sound indicates that a menu is now active and canbe manipulated. This sound may be described as a fade-in sound thatterminates with a snap-in-type sound. In addition, the AIM 402 canpresent a short vibration cue that confirms the user's intention toproduce a change in state. Further still, the AIM 402 may invoke averbal cue that announces, for example, “menu release to dismiss.” Thatcue informs the user that the menu is active, and that the user candeactivate the menu by releasing his or her finger from the marker item.

Menu-Deactivation. This sound indicates the menu has been deactivated,and thus can no longer be manipulated without reactivating it. The soundmay be described as a fade-away-type sound.

Unsupported-Gesture. The AIM 402 may play this short sound thatindicates that a gesture was recognized but that it is otherwise invalid(e.g., because it is currently not supported). This sound may resemble athud, followed by a soft two-tone access-denied-style notification. TheAIM 402 may play a more specific Cannot-Go-Back sound to indicate that aback gesture has been performed, but it cannot be performed, e.g.,because it is not possible to go further back.

Change-In-Menu-Item-Selection. This sound indicates that the user hasmoved to a menu item in a menu. The tone of this sound depends on theposition of the menu item in the menu, e.g., where the tone may increaseas the user moves up the list, and decrease as the user moves down thelist. Further, in one implementation, the AIM 402 can present differentscales when traversing the first grouping of the menu items (associatedwith the context-specific items), compared to the second grouping ofmenu items (associated with the global items). In both cases, the usermay perceive the sounds that are produced as similar to the running of ahand over the keys of a piano; but each sound in this case may beshorter that a piano tone, and similar to the pop of a bubble.

Further, the AIM 402 can present a vibration cue upon traversing eachmenu item. The user may experience the vibration cues as similar torunning a hand over a bumpy surface, with each bump representing adifferent menu item. A user can consciously or subconsciously count thebumps to quickly get a general idea of his or her position within thelist. Further, the AIM 402 can present an audio message upon advancingto a menu item that informs the user that the menu item is in focus andwill be selected upon release of the user's finger. For instance, theAIM 402 can announce the menu item by giving its number in the list, andthen announcing a description of the item.

Confirm-Menu-Selection. This short sound indicates when an action hasresulted in a change to the state. It may be implemented as a short andprecise sound with a flourish at the end. For instance, the sound mayresemble a bleep followed by a travelling grating sound that fades outtowards its end. The AIM 402 may also execute a short vibration cue thatconfirms the select action. Further, the AIM 402 can play a verbal cuethat confirms that the item has been selected.

Show-Dialog-Menu. This short sound indicates that an overlay menu is nowbeing presented on top of other content. This sound may resemble afade-in ascending-type sound. The AIM 402 can also provide anaccompanying short vibration cue, which further confirms that a changein state has taken place. Further, the AIM 402 can play a verbal cuethat states that the overlay menu is now being presented.

Close-Dialog-Menu. This short sound indicates that a dialog (e.g., anoverlay menu) has been closed and that the focus has returned to aprevious context. The sound may resemble a fade-out descending-typesound.

Context-Switch. This short and distinct sound indicates that the stateof the current context menu has transitioned to present new content,e.g., because the user has advanced to a new journey step, etc. Thesound may resemble a bleep followed by a subtle turnstile noise.

The above-described sounds are representative, and not exhaustive of thefull suite of sounds that the AIM 402 may provide. The followingadditional sounds, for instance, may be triggered upon certain actions,performed by the user, while interacting with the AIM 402: a) aTo-Actions sound to indicate that a menu of actions is presented, basedon a selection of a menu item made in a prior menu; b) a From-Actionssound to indicate that an active menu is presented in response to thereturn from a menu of actions; c) an Actions-Unavailable sound toindicate that an instruction to present actions for a particular menuhas been recognized, but that there are no actions associated with theparticular menu; d) an Item-Selectable sound that indicates that a menuitem has been successfully marked for selection upon release of theuser's finger; e) an Item-Non-Selectable sound to indicate that an itemthat was previously marked for selection-upon-release has now beensuccessfully un-selected, and thus will no longer be selected uponrelease of the user's finger; f) an Item-Not-Selectable sound toindicate that a gesture to mark an item for selection has beenrecognized, but the gesture is not applicable to the item underconsideration; g) a Select-But-No-Change sound to indicate that aselection has been made, but that no change in focus is appropriate; h)a Back-Success sound to indicate that a back gesture was recognized andthe back action has been invoked; i) a Switch-To-Tabs sound to indicatethat a switch-to-tabs gesture has been successfully recognized and thatthe tabs menu is now being presented; j) a Switch-From-Tabs sound toindicate that a switch-from-tabs gesture has been recognized and theprior menu, that was presented prior to switching to the tabs menu, hasbeen restored; k) a Start-Listening sound to indicate that the SI module302 is now listening for a user's voice commands; l) a Cancel-Listeningsound to indicate that the voice recognition mode has now been canceled;m) a Finished-Processing sound to indicate that the application hascompleted receiving and processing a voice input; n) an Action-Takensound to indicate that an action has been taken as a result of avoice-based input, and so on. Further, the AIM 402 can also present anytype of haptic cues that will accompany any of the sounds describedabove.

The following illustrative sounds may be triggered upon certain eventsthat may occur in the course of the user's interaction with the headset108: a) a Button-Tap sound (played prior to performing an action) toindicate that a tap gesture has been recognized on a headset button; b)a Button-Press sound (played prior to performing any action) to indicatethat a press gesture has been recognized on a headset button; c) aButtons-Held sound (played prior to performing an action) that indicatesthat a hold gesture has been recognized on a headset button; d) aCancel-Action sound that indicates that any previous action, invokedfrom a headset button, has been cancelled, e.g., for a request to stopannouncing contextual IOIs as part of the orientation action, and so on.Further, the AIM 402 can also present any type of haptic cues that willaccompany any of the sounds described above.

The following illustrative sounds may be triggered upon certain eventsthat may occur during the course of a journey: a) a Started-Journeysound to indicate that a journey has been started and that navigation isnow in progress; b) a Mode-Switch sound that indicates that the user hasswitched his or her mode of transport (e.g., from walking to train,etc.); c) a Beat sound to directionally indicate the next point alongthe journey that the user is trying to get to as part of navigation; d)a Warning sound to indicate that warning information is about to be readaloud, played to heighten the user's awareness and give the user time totune-in for that information; e) a Waypoint-Reached sound to indicatethat the user has reached a journey waypoint and that navigationinformation is about to be read aloud, played to heighten the user'sawareness and give the user time to tune-in for that information; f) anApproaching Waypoint sound to indicate that the user is approaching awaypoint and that navigation information is about to be read aloud,played to heighten the user's awareness and give the user time totune-in for that information; g) a Journey-Update sound that is playedto indicate that information regarding a change to the current journeyis about to be read aloud, played to heighten the user's awareness andgive the user time to tune-in for that information; h) aContextual-Information sound that indicates that contextual informationis about to be read aloud; i) a Further-Information sound that indicatesthat additional information is about to be read aloud, and so on.Further, the AIM 402 can also present any type of haptic cues that willaccompany any of the sounds described above.

Generally, the above sounds and haptic cues further promote the goal ofproviding useful information to the user as the user interacts with hisor her environment, without unduly distracting the user. For example,the sounds and haptic cues that the user hears while navigating a menuallow the user to interact with the menu without diverting his or herattention from the environment.

C.3. Voice Recognition Mode

The system 102 supports a voice-recognition mode in which the user mayissue commands to the system 102 via spoken instructions, e.g., inaddition to manually interacting with the headset 108 and/or user device106. Or the system 102 can use the voice-recognition mode as the solemode in which the user interacts with the system 102. To performrecognition of the user's voice commands, the system 102 may rely onpre-existing voice recognition technology (e.g., the CORTANA systemprovided by MICROSOFT Corporation), and/or native voice recognitiontechnology. FIG. 3 illustrates the voice recognition technology as thespeech processing engine 328.

As mentioned above, the user can invoke the voice-recognition mode viaan appropriate command issued through the headset 108 and/or the menusof the user device 106. In other cases, the user device 106 may alreadybe interacting with a voice recognition system (e.g., CORTANA) as partof its default manner of operation, but not in the context of performingnavigation. Here, the user may interact with the SI module 302 in thevoice-recognition mode by issuing a spoken command that is preceded bythe name of the navigation application, e.g., by speaking the command,“Soundscape, start explore mode.” If the user has already activated thevoice-recognition mode in the context of the navigation application, theuser can just say, “Start explore mode,” or the like.

The following list identifies illustrative operations that may beinitiated in voice mode: a) the user may open an existing (saved)journey; b) the user may activate or reactive the orientation mode; c)the user may activate or deactivate the explore mode; d) the user mayask for more information about a particular topic; e) the user mayrequest the cessation of all (or some) spoken messages; f) the user maymake various parameter settings; g) the user may save journeyinformation, and so on.

Illustrative commands may correspond to: a) “Create a route todestination [x], leaving at time [y], using only public transport [z]”;“Find the closest coffee shop”; c) “Get me to a train station”; d) “Whatis the time?”; e) “Increase volume”; f) “Remove restaurants from itemsof interest,” etc.

The speech processing engine 328 may sometimes encounter a situation inwhich it understands the user's command, but it determines that thecommand omits one or more items of necessary information. In response,the speech processing engine 328 can ask the user to supply the missingitems of information.

In other cases, the speech processing engine 328 may encounter asituation in which it does not understand the user's command. In thosecases, the speech processing engine 328 can ask the user to rephrase thecommand. If that is not successful, the speech processing engine 328 maypresent proposed interpretations of the user's utterance to the user(e.g., based on the detection of keywords in the user's command), andthen ask the user whether any of the interpretations are correct. Ifthat is not successful, the speech processing engine 328 may invite theuser to enter an equivalent command via the headset 108 and/or the userdevice 106.

FIG. 38 shows a process 3802 which summarizes one manner of operation ofthe AIM 402 of FIGS. 4 and 5. In block 3804, the AIM 402 detects a firstgesture, performed by the user, corresponding to an instruction toactivate a menu in an associated workspace. The associated workspacecorresponds to one of a plurality of workspaces, and the associatedworkspace has a determined spatial relationship with respect to otherworkspaces in the plurality of workspaces. The first gesture, forexample, may correspond to the above-described tap-and-hold gesturedescribed with reference to FIG. 24. In block 3206, the AIM 402activates the menu in response to the first gesture.

In block 3808, the AIM 402 detects a second gesture, performed by auser, corresponding to an instruction to advance to a particular menuitem, among a collection of menu items in the menu. The second gesture,for example, may correspond to any of the types of scrolling or panninggestures described above (e.g., with reference to FIGS. 25 and 31). Inblock 3810, the AIM 402 advances to the particular menu item in responseto the second gesture.

In block 3812, the AIM 402 detects a third gesture, performed by a user,corresponding to an instruction to select a particular menu item. Thethird gesture, for example, may correspond to the type of releasegesture shown in FIG. 25 or the type of pull-to-the-side gesture show inFIG. 32. In block 3814, the AIM 402 performs an operation in response tothe third gesture. The operation may correspond to the invocation of anaction, the setting of a parameter, the presentation of information,etc.

In summary, the above features allow the user to safely and efficientlymove through his or her environment. For example, the workspaces providea user-friendly way of surfacing the most relevant information to theuser as the user moves through the environment. Further, some of thefeatures allow the user to interact with a user interface presentationwithout directing visual attention to that presentation, and/or withouthaving to perform cumbersome and complex gestures that divert the user'sattention from the physical task of interacting with the environment.For instance, the user may perform some of the gestures with a singlehand without looking at the user interface presentation.

D. Facilitating Interaction Between Users and their Environments UsingSounds

As described in the introductory Section A, the sound generation module414 (of FIG. 4) can generate non-three-dimensional (non-spatial) soundsand three-dimensional (spatial) sounds. A three-dimensional sound is asound which the user perceives as emanating from at least one locationin physical space, even though it has no actual origin in physicalspace. For specifically, FIG. 39 demonstrates the use ofthree-dimensional audio information to create a perception of soundwhich emanates from a particular location within space. FIG. 40demonstrates the use of three-dimensional audio information to create aperception of sound that moves across a series of locations withinspace.

In one implementation, the sound generation module 414 can produce athree-dimensional sound using a library of Head-Related TransferFunctions (HRTFs), e.g., as provided in the data store 416. An HRTFmodels the anatomical features a person's face that have aperson-specific bearing on the manner in which that person perceives theorigin of sound in his or her environment. To produce athree-dimensional sound, the sound generation module 414 can choose anHRTF that is appropriate for a particular person, and for a particularsound location in space (relative to the person's location). The soundgeneration module 414 can then apply that HRTF to modify anon-three-dimensional sound (e.g., a flat or non-spatial sound), toproduce the three-dimensional sound, e.g., by convolving the non-spatialsound with the chosen HRTF.

More specifically, in one case, the sound generation module 414 canreceive input information which describes: (a) the non-three-dimensionalsound to be played; (b) the location in space at which the sound is tobe perceived as originating; and (c) (optionally) the identity of theuser. The input information can describe the non-three-dimensional soundby providing reference information which identifies audio information inthe data store 416, and/or may provide the audio information itself. Forexample, the non-three-dimensional sound may correspond to a telltaletone and/or a spoken message. Then, for each ear of the user, the soundgeneration module 414 can: (a) identify an HRTF associated with thelocation, for the particular user under consideration; and (b) apply theHRTF to the non-three-dimensional sound to produce the three-dimensionalsound. The input information that is provided to the sound generationmodule 414 may originate from one or more of the other modules of the SImodule 302 described below, e.g., corresponding to the path guidancemodule 420, the relevant information determination (RID) module 426, theexploration module 430, the orientation module 432, etc.

The sound generation module 414 performs the above functions withrespect to a library of HRTFs provided in a data store 416. In oneimplementation, the SI module 302 can store HRTFs in the data store 416that have been prepared for particular individuals (and which thus takeinto account the particular anatomical features of these people). Forexample, the HRTFs, for each person, can be obtained by measuring thephysical characteristics of each person's face. That task, in turn, canbe performed in manual fashion (e.g., by using physicaldistance-measuring tools), by using a depth camera (e.g., by using theKINECT system provided by MICROSOFT Corporation of Redmond, Wash.), andso on. More specifically, the set of HRTFs that is generated, for aparticular person, includes an HRTF for each location of sound in space(with respect to the position of the user's head), and for each of theuser's two ears. Different implementations may entrust differentindividuals to create these HRTFs (e.g., acoustic engineers, systemadministers, end users, etc.).

In a second implementation, the sound generation module 414 can storeseparate groups of HRTFs that perform well for different respectivegroups of people. The system 102 can then invite an end user to choosethe set of HRTFs that is perceived as producing the most realisticthree-dimensional sounds. For instance, a user may make this choice in aset-up phase, by asking the system 102 to produce the samethree-dimensional sound(s) using different HRTFs; the user may choosethe sound(s) (and the corresponding HRTF) that produce the mostdesirable result. In a third implementation, the sound generation module414 can store a single set of HRTFs that have proven suitable for alarge population of users, even though these HRTFs are not customizedfor any particular person or group of people. Still other techniques maybe used to generate and/or select suitable HRTFs.

In one implementation, the sound generation module 414 generatesthree-dimensional sounds that convey wideband audio information.Wideband audio information is audio information that includes a widedistribution of audio frequencies, e.g., in one implementation, in therange of 300 Hz to 3.4 kHz. Further, the individual three-dimensionalsounds are expressive in nature to further emphasize theirdirectionality. For example, one such three-dimensional sound mayresemble a gust of wind blowing from left to right or from left toright. Other properties of the audio information (besides its HRTF-baseddirectionality) may contribute to its expressiveness, such as byproviding variations in volume, tone, reverberation, looping frequency(for repeating sounds), etc. All of these factors contribute to arealistic perception that sound is originating from a particularlocation (or locations) in physical space.

As also described in Section A, different modules may leverage the useof three-dimensional sounds for different purposes and in differentmodes of operation. These modules include the path guidance module 420,the RID module 426, the exploration module 420, the orientation module432, etc. To simplify description, each such module is sometimesdescribed as producing three-dimensional sounds. In actuality, eachmodule produces the sound by generating the above-described inputinformation, which it feeds to the sound generation module 414; thesound generation module 414 then uses the input information to producethe actual three-dimensional sounds.

With reference to FIG. 41, first consider the operation of the pathguidance module 420. As already described, the path guidance module 420determines the current direction in which the user is headed. The pathguidance module 420 can make this assessment based on any of theinformation provided by the system's orientation determinationmechanism(s), motion determination mechanism(s), position determinationmechanism(s), and so on. In one case, for instance, the path guidancemodule 420 can determine the direction in which the user's head ispointed (e.g., based on an orientation determination mechanism in theheadset 108), and then use this information as a proxy for the directionin which the user is presumed to be headed. Alternatively, or inaddition, the path guidance module 420 can determine a series oflocations that the user has recently traversed. The path guidance module420 can use this information to project a direction that the userappears to be headed. Alternatively, or in addition, the path guidancemodule 420 can determine the direction in which the user is purposelypointing his or her user device 106.

The path guidance module 420 can also determine the desired direction ofthe user. The path guidance module 420 can determine the desireddirection based at least the current location of the user and thelocation of the next waypoint. The path guidance module 420 can alsotake into consideration map information when determining the desireddirection. For example, the map information may reveal that the user'spath is restricted in various ways, e.g., by the course of a roadway,obstacles of any type etc. The path guidance module 420 can use all ofthis information to determine the direction that the user should beheaded to ultimately place the user on course to reach the nextwaypoint, in the most efficient manner possible.

Based on the actual direction and the desired direction, the pathguidance module 420 can then determine the extent to which the user maybe deviating from the desired direction, e.g., to provide deviationinformation. The path guidance module 420 can then use the deviationinformation to generate a three-dimensional sound that will have theeffect of steering the user in a desired direction, e.g., by feeding thedeviation information to the sound generation module 414 as part of theinput information.

To clarify the above description, consider the illustration of FIG. 41.Here, the user is located at a current location 4102. The next waypoint(w) is located at target location 4104. The user's current (actual)direction is denoted as direction 4106. The user's desired direction4108 corresponds to that vector which connects the current location 4102of the user with the target location 4104, although, as noted above,this need not be the case in all situations (e.g., due to an obstructionwhich blocks a direct route from the current location 4102 to the targetdestination 4104). The deviation information may correspond to theangular difference between the current direction 4106 and the desireddirection 4108.

In response to the above determinations, the path guidance module 420can produce a three-dimensional sound (using the sound generation module414) which appears to originate from the location 4110. In other words,the user is currently headed off to the left of the desired direction4108. The user will hear the directional cue emanating off to the right.The user will perceive that cue as urging the user to veer towards theright to correct the direction of his or her path. If the user startsmoving in a direction that is too far to the right of the desireddirection 4108, the path guidance module 420 may produce athree-dimensional sound (using the sound generation module 414) off tothe left of the user. The user may continue on, essentially following orpursuing the direction of the perceived three-dimensional sound. Whenthe user is headed in the correct direction, the three-dimensional soundwill be perceived as emanating directly in front of the user; the userthen heads straight for that sound.

As noted above, the path guidance module 420 can use the soundgeneration module 414 to create a three-dimensional sound that isperiodic in nature, e.g., corresponding to a beat sound. For instance,the beat sound may correspond to a repeating single tone or a two-toneor n-tone clip-clop sound, and so on. In addition, or alternatively, thethree-dimensional sound can appear to move in the direction that theuser is being urged to move, e.g., from left to right, or right to left,etc.

Moreover, the path guidance module 420 can provide input information tothe sound generation module 414 which has the effect of varying one ormore audio characteristics of the beat sound, depending on an extent towhich the user's actual current direction 4106 deviates from the desireddirection 4108. For instance, the path guidance module 420 (inconjunction with the sound generation module 414) can generate a beatsound having a first tone and/or a first looping frequency when theuser's current deviation from the ideal path is within a firstthreshold. The path guidance module 420 (in conjunction with the soundgeneration module 414) can generate a beat sound having a second toneand/or a second looping frequency when the user's current deviation isoutside the first threshold, but within a second threshold, and so on.The path guidance module 420 can classify the user's deviation into anynumber of such deviation categories, each being associated with aparticular beat sound.

In the particular example of FIG. 41, the path guidance module 420defines a first range 4112 of deviations from the desired direction 4108and a second range 4114 of deviations from the desired direction 4108,the second range 4114 being larger than the first range 4112. Thecurrent direction 4106 falls within the second range 4114, and thereforethe path guidance module 420 (in conjunction with the sound generationmodule 414) will generate a beat sound that is appropriate for thisrange 4114.

In actual practice, the path guidance module 420 is useful in manycircumstances, but may be perceived as most useful in two circumstances.In a first case, a user may reach an intersection at which he or she isexpected to turn in a particular direction, among a fixed set ofavailable options. Or the user may reach a fork in the road in which heis expected to choose one of the possible paths. The path guidancemodule 420 (in conjunction with the sound generation module 414) canprovide unambiguous guidance to the user by playing a three-dimensionalbeat sound that the user perceives as resolving his or her navigationchoices. For example, when standing in an intersection, the user willhear the beat sound leading him or her to the left or right (forexample). When standing at a fork in the road, the user will perceivethe beat sound as leading him or her down the correct fork in the road.

In a second scenario, the user is moving across a more open space inwhich he or she is given more latitude to gradually veer off course(compared to the case in which the user is walking down a well-definedroad or path). Here, the path guidance module 420 may give the userincremental nudges in the manner illustrated in FIG. 41 to keep the useron a proper heading.

In another use scenario, the relevant information determination (RID)module 426 can determine, at each point along a user's path, and foreach corresponding current context, the location of relevant items ofinterest (IOIs), such as contextual IOIs. The RID module 426 can thenuse the sound generation module 414 to generate a three-dimensionalsound for each IOI. The user will perceive that sound as originatingfrom the actual location of the corresponding physical entity inphysical space, that is, assuming that the IOI corresponds to a physicalentity having a discrete physical location in space. For example, theRID module 426 (in conjunction with the sound generation module 414) cangenerate the following audio information for a particular IOI that has acounterpart physical entity in the user's vicinity: [3D sound], “Jane'sCoffee Shop, restaurant, 30 feet ahead.” In other words, in this case,the audio information includes a three-dimensional sound, followedshortly thereafter by a spoken message. The user will perceive thethree-dimensional preliminary sound as emanating from a physicallocation of Jane's Coffee Shop. The spoken message may also be presentedas a three-dimensional sound (although it need not be). The spokenmessage could alternatively include additional (or fewer) fields ofinformation, or could be omitted entirely.

In other cases, an IOI may have a general origin in space, rather than apinpoint location. For example the IOI may correspond to anadvertisement that is associated with a general area that, in turn,corresponds to a bus stop. The RID module 426 (in conjunction with thesound generation module 414) can generate a sound for that IOI that isperceived by the user as emanating from the general area of the busstop, or from the center of that area, etc. That type of IOI may beconsidered virtual insofar as it is not intended to describe the busstop, but operates as an audio attribute of the bus stop.

In another implementation, the audio information can also include averbal cue which specifies the directional location of the IOI, relativeto the user. That directional cue can be expressed broadly (e.g., byspecifying that Jane's Coffee Shop is ahead of the user and to theright), or more narrowly (e.g., by specifying that Jane's Coffee Shop islocated at 10 o'clock, relative to the current direction of the user).In addition, or alternatively, the RID module 426 (in conjunction withthe sound generation module 414) can provide different three-dimensionalsounds for different respective categories of IOIs, e.g., by playing afirst type of sound for restaurants and a second type of sound forrestroom facilities, etc.

Both the exploration module 430 and the orientation module 432 leveragethe above-described behavior of the RID module 426, e.g., in the exploremode and the orientation mode, respectively. As described in Section A,the user may enter a command to expressly activate and deactivate thesemodes, e.g., via manual interaction with the headset 108 and/or the userdevice 106, and/or via a voice instruction.

More specifically, FIG. 42 shows the illustrative behavior of theexploration module 430, operating in the explore mode. In this mode,once the explore mode is activated by the user, the exploration module430 determines the current location of the user and the current focus ofinterest of the user. For example, the exploration module 430 can useany position determination mechanism(s) on the headset 108 and/or on theuser device 106 to determine the location of the user at the presenttime (e.g., using a GPS sensor, etc.). The exploration module 430 canuse any orientation mechanism(s) on the headset 108 and/or on the userdevice 106 to determine the direction at which the user appears to beorienting his body. For example, the orientation mechanism(s) on theheadset 108 can determine the direction at which the user's head ispointed, which can be used as a proxy-indication of the user's focus ofinterest. Alternatively, or in addition, the system 102 may instruct theuser to point the user device 106 in the direction that matches his orher focus of interest; in that case, the orientation mechanism(s) of theuser device 106 will correctly reflect the user's intended focus ofinterest. Still other techniques can be used to assess the user'spresumed focus of interest.

Next, the exploration module 430 defines a search space of interest. Inone case, the exploration module 430 may define the search space as avolume of any shape that is centered on the user's presumed focus ofinterest. For example, the volume may correspond to a wedge-shape volumewhich originates from the user's current location and which is bisectedby the user's presumed focus of interest. System-defined and/oruser-defined parameter settings may specify the depth of the volume, theangular span of the volume, the width of the volume, etc.

Next, the exploration module 430 may use the RID module 416 to searchthe volume for the presence of IOIs of interest, such as, but notlimited to, contextual IOIs. The RID module 416 can perform this task byconsulting one or more data stores which provide information about IOIs,together with any parameter settings which define the user's particularinterests (for example, the user may have indicated that he or she is,by default, interested in restaurant information, with the exclusion ofcoffee shop information). In some cases, an IOI may be associated withthe search space because it has a counterpart physical entity that isphysically located in the search space. In other cases, an IOI may beassociated with the search space because of some other nexus establishedin any other way. For example, an administrator (or an end user himselfor herself) can manually specify, in advance, that the area near theexit of a subway station is to be associated with a weather report IOI,and so on.

Finally, the exploration module 430 can generate three-dimensional audiomessages which announce the presence of the IOIs that have beenidentified. The exploration module 430 can generate these messages inany manner described above by leveraging the sound generation module414. For example, as described above, the audio information maycorrespond to an introductory sound, followed by a spoken message whichannounces the IOI. The exploration module 430 (in conjunction with thesound generation module 414) can read off these IOIs in an order, suchas in clockwise order across the volume, or counterclockwise order,and/or by increasing or decreasing distance relative the location of theuser, etc.

FIG. 42 clarifies the manner of operation described above. Here, theuser is presently located at a current location 4202. The user's currentdirection of attention is indicated by the dashed line 4204. The searchvolume is defined by the contour 4206, which is bisected by the user'sdirection of attention. That search volume defines an angular swath, andmay also have a prescribed height (not shown), such as 30 feet. Overall,the search volume resembles a wedge. The exploration module 430 uses theRID module 426 to find three contextual IOIs in the search volume. Theexploration module 430 (in conjunction with the sound generation module414) announces these contextual IOIs by three-dimensional sounds whichappear to originate from the locations of the physical entitiesassociated with the contextual IOIs.

Overall, the user may interact with the exploration module 430 byrotating his direction of attention, by pivoting about his currentlocation, in stages. At each stage, the user may wait to hear the IOIsthat lie within (or are otherwise associated with) the search volumethus defined by his current presumed attention of focus, prior to movingon to a new focus of attention. For example, in one case, the swathencompassed by a user's direction of interest is 45 degrees. The usercould conduct a complete inventory of IOIs around him by turning to thenortheast, the southeast, the southwest, and the northwest, insuccession. At each orientation, the user will hear the IOIs that areencompasses in that quadrant of interest.

According to another illustrative feature, the user may select an IOI inany manner after hearing it announced. For example, the user may issuethe voice instruction “take me there” or “more information” afterhearing about Jane's coffee shop. Or the user may select the IOI byturning his or her focus of attention towards the perceived location ofthe IOI (e.g., based on the perceived location of the three-dimensionalsound which announces this IOI). The user may perform this task, forinstance, by turning his head or body or user device 106 directlytowards the perceived location of the IOI. Or the user may select theIOI via an appropriate menu provided by the user device 106. Or the usermay make an appropriate selection via a headset button after hearing theIOI being announced, etc. In response to the selection of the IOI, inwhatever manner made, the SI module 302 can present additionalinformation regarding the IOI, or provide instructions on how to reachthe IOI, etc. The SI module 302 can also confirm the user's selection byincreasing the volume of the three-dimensional sound that announces thepresence of the chosen IOI, or by otherwise making that sound moreprominent in the user's experience.

Finally, the orientation module 432 operates by determining the currentlocation of the user in the manner described above. It then defines athree-dimensional volume around the user. For example, thethree-dimensional volume may correspond to a cylinder or sphere or boxor rectangle (for example), with the user located at the center of thevolume. The orientation module 432 then uses the RID module 426 toidentify the set of IOIs that exist within the volume, if any. Theorientation module 432 then uses the sound generation module 414 to readoff the IOIs in any order.

For example, in FIG. 43, the orientation module 432 defines athree-dimensional volume having the shape of a cylinder, with the userpositioned at its center at a current location 4302. The orientationmodule 432 uses the RID module 426 to identify the IOIs that lie withinthe cylinder or are otherwise associated with the space defined by thecylinder. Then the orientation module 432 reads off the IOIs in anyorder. For example, consider a user who visits a mall having threefloors. Assume that the user is standing in the open-air atrium on thefirst floor of the mall. The orientation module 432 can read off theIOIs that it finds on a floor-by-floor basis, e.g., floor z₁, floor z₂,and then floor z₃. The orientation module 432 can announce the IOIs oneach floor in any order, such as a clockwise order, a counterclockwiseorder, etc. The user may then optionally select any IOI in any of theways described above.

As closing comment, the above explanation set forth the use ofthree-dimensional sounds to announce the presence of contextual IOIs,such as restaurants, etc. But the SI module 302 can usethree-dimensional sounds to announce the presence of any types of IOIs,such as warning IOIs and journey IOIs. For example, the SI module 302can generate a warning regarding a pothole in the road that appears toemanate from the location of the pothole. As another clarification, theSI module 302 can also deliver many types of IOIs in a flat ornon-spatial manner. For example, the SI module 302 can produce anotherwarning IOI that has no directivity to generally notify the user thatthe road on which he is currently traveling is slick due to rain.

Further, the above description was predicated on the use of sound thatthe user perceives as originating from locations in physical space. Inaddition, or alternatively, the haptic cue generation module 418 cangenerate vibration cues which convey directional information. Forexample, the headset 108 can include two or more vibration mechanisms,e.g., a first vibration mechanism on the left side of its frame 602 anda second vibration mechanism on the right side of its frame 602. Thehaptic cue generation module 418 can activate either the first or secondvibration mechanism to provide instructions to the user to turn left orright, respectively. This type of instruction can include additionalgradations by including additional vibration mechanisms. The same effectcan be achieved by activating different vibration mechanisms on the userdevice 106, e.g., by activating a left-side vibration mechanism toprovide a cue to turn left, and a right-side vibration mechanism as acue to right. The vibration mechanisms can be coupled to yet otherdevices or parts of the user's body.

FIG. 44 shows a process 4402 that describes a use of three-dimensionalsounds to guide the user in navigating along a desired route. In block4404, the SI module 302 determines a current location of a user along aroute, within the space. In block 4406, the SI module 302 generates athree-dimensional sound. The three-dimensional sound creates aperception, by the user, that the three-dimensional sound emanates fromat least one particular location within the space. In block 4408, the SImodule 302 delivers the three-dimensional sound to the user. Thethree-dimensional sound assists the user in navigating along the route.

FIG. 45 shows a process 4502 that describes one manner in which the pathguidance module 420 may use the three-dimensional sounds to guide theuser along a desired route. In block 4504, the path guidance module 420determines a current direction in which the user is headed. In block4506, the path guidance module 420 identifies a desired direction inwhich the user is expected to be headed. In block 4508, the pathguidance module 420 determines a difference between the currentdirection and the desired direction, to provide deviation information.In block 4510, the path guidance module 420 uses the sound generationmodule 414 to generate a periodic three-dimensional sound (e.g., a beatsound), based on the deviation information. That three-dimensional sounddirects the user towards the desired direction. The three-dimensionalsound may have one or more properties (such as tone) which depend on theextent to which the user has deviated from the desired direction.

FIG. 46 shows a process 4602 that describes one manner in which the SImodule 302 can use three-dimensional sounds to identify and announceIOIs. In block 4604, the SI module 302 determines a set of items ofinterest (IOIs), each item of interest corresponding to an entity orevent or piece of information that is relevant to the user in a currentcontext. In block 4606, the SI module 302 generates a three-dimensionalsound for each item of interest. The user will perceive the sound asoriginating from the same physical location (or locations) that, inturn, is associated with the item of interest, e.g., by perceiving asound associated with a store as emanating from the location of thestore.

In one case, the SI module 302 can apply the process 4602 as abackground service as the user traverses a route. For example, the RIDmodule 426 can alert the user to the existence of IOIs when the userdraws sufficiently close to the locations associated with these IOIs, asgoverned by any distance-based parameter setting.

In another scenario, as indicated in block 4608, the exploration module430 may apply the above general process 4602 by using the RID module 426to identify a set of IOIs that are associated with a subspace to whichan attention of the user is current directed, or presumed to becurrently directed. In another scenario, as indicated in block 4610, theorientation module 432 can apply the above general process 4602 by usingthe RID module 416 to identify a set of IOIs that are associated with anentire space around the user at the current time, without reference tothe user's current focus of interest (because the user is now interestedin the complete volume of space around him or her at the moment). Theuser can invoke and suspend the readout of the exploration module 430 orthe orientation module 432 by issuing appropriate instructions, e.g.,via the headset 108 or the user device 106.

As a final topic, the above examples were based on the simplifyingassumption that the position of the virtual sound-producing entity(e.g., the virtual sound source) in space is stable relative to thelocation of the listener, at least over the course of the delivery ofthe three-dimensional sound. This may be a valid assumption in manycases, but it may not hold true for all situations.

For example, consider the following scenarios. In a first case, a usermay listen to a three-dimensional sound that describes (or otherwiserelates to) a fixed-position entity, as the user passes the entity in avehicle of any type (e.g., on a train). For example, the user may listento a message that describes Seattle's Space Needle tower as the user istraveling as a passenger in a car, in any direction relative to theSpace Needle. In a second case, the may listen to a three-dimensionalsound that describes (or otherwise relates to) a fixed-position entity,as the user stands still, but nevertheless moves his or her head about,such that the position of the user's left and right ears are changingrelative to the location of the fixed-position entity. In a third case,a user may listen to a three-dimensional sound that describes (orotherwise relates to) an in-motion entity, relative to a fixed positionof the listener. For example, the user may listen to a message thatannounces the arrival of an airplane that is moving down a runway, whilethe user remains at a fixed position in the airport's terminal. In afourth case, both the location of an IOI and the user may be in motionduring the delivery of the audio information.

Further note that an IOI need not always correspond to a physicalentity, such as the Space Needle or a moving airplane. For example, amoving “virtual” IOI may correspond to a virtual billboard whichdelivers a spatial message that the user perceives as moving down thestreet towards the entrance of a nightclub, enticing the user to enterthat establishment. Further, everything that is set forth herein withrespect to sounds associated with IOIs applies equally to other sounds,such as the periodic beat sound.

To address any of the above situations, the SI module 302 candynamically update the three-dimensional audio information that it isproducing over the course of its delivery, to reflect, at each instanceof time, the relative position between the user's head and the IOI(e.g., the location of the Space Needle or the moving airplane, in theexamples cited above). The SI module 302 can perform this task indifferent ways. In one non-limiting approach, the SI module 302 canperform the following operations in an iterative process over thedelivery of the audio information: (a) determine the position of theuser's head relative to the IOI, to provide relative positioninformation; (b) select an HRTF (per ear) based on the relative positioninformation; (c) convolve whatever audio information is to be deliveredat the current time by the selected HRTF, to produce a three-dimensionalsound that is suitable based on the relative position information.

To cite one specific example, consider a message that takes five secondsto deliver to the user. The SI module 302 can determine, for each secondof that delivery, the relative position between the user's head and theIOI. The SI module 302 can use that information to produce athree-dimensional sound for each second of the message's delivery thatis based on the relative position between the user's head and the IOI atthat instance of time.

More generally, the above iterative processing can be performed atdifferent update rates. The update rate may depend on the rate at whichthe relative position between the listener and the IOI changes. That is,the SI module 302 can update the HRTFs at a relatively quick rate forlarge relative changes to the relative position information, and at aslower rate for slower changes. The update rate may also take intoconsideration the amount of processing resources that are available inthe system 102 to update the three-dimensional sound information.

The SI module 302 can also apply various techniques that are designed toexpedite the above dynamic processing (which may represent a largeprocessing burden on its resources). For example, in some cases, thesystem 102 can pre-compute a dynamic sound for at least one predictedtrajectory; each such trajectory defines some type of expected relativemovement between a listener and the IOI, due to: (a) the movement of thelistener in space; or (b) the movement of the IOI in space; or (c) themovement of both the listener and the IOI. For example, assume that atrain passes by an IOI at generally the same speed each day. In thiscase, it is possible to pre-calculate a three-dimensional sound (perear) that is predicated on a dynamically-changing HRTF. Thatthree-dimensional sound takes into account a presumed progression of theuser's head as the train moves down the tracks, and may be based on thesimplifying assumption that the user's head has a predetermined fixedorientation as the user moves down the tracks. The SI module 302 canlaunch that three-dimensional sound when it detects that a listener'sposition reaches a predetermined triggering location relative to theIOI, such as a location along the tracks that is a predetermineddistance from the IOI.

The above manner of operation can be expanded to account for differentscenarios. For example, the system 102 can pre-compute different dynamicthree-dimensional sounds to take account for different train speeds,different fixed head orientations (e.g., indicating whether the user islooking straight ahead during the delivery of the message, or lookingout the window, etc.), and so on.

The above situation can also be extended to a more general setting inwhich the movement between the user and an IOI can be decomposed into anumber of possibilities, allowing for the pre-calculation of respectivedynamic three-dimensional sounds for these possibilities. The SI module302 can then launch any pre-calculated sound when it determines that theuser's current context matches one of the predetermined possibilities.

In another case, the SI module 302 can dynamically perform processing topredict the movement of the user relative to the location of an IOI,e.g., on the basis of the user's current heading. The SI module 302 canthen dynamically pre-compute a three-dimensional sound based on aprojected trajectory of the user relative to the IOI.

Still other techniques can be used to expedite the computation ofdynamically-changing three-dimensional sounds. In some implementations,for instance, the SI module 302 can draw on the enhanced processingcapabilities of the remote processing resources 112, e.g., by leveragingparallel processing performed by those resources 112.

In summary, the above features contribute to the above-stated goal ofallowing the user to efficiently, safely, and enjoyably move through hisor her environment. For instance, the features provide the user withdifferent categories of information as the user traverses theenvironment, using different modes of presentation. The functionalitypresents this information to the user in a user-friendly manner thatavoids overwhelming the user with too much information at any giventime. The use of three-dimensional sounds further enhances the abilityof the user to understand the nexus between the information that isprovided and objects, regions, and events in the environment.

E. Use of Beacons for Assistance to Users in Interacting with theirEnvironments

The following section provides additional information regarding theoperation of the beacon-based guidance module 422. To repeat, thebeacon-based guidance module 422 provides a strategy for guiding theuser through a space by leveraging the use of beacons having definedrespective ranges. The space may correspond to an indoor space, e.g., aspace defined by the interior of one or more buildings or structures ofany type. Alternatively, or in addition, the space may correspond to anoutdoors space.

The beacons may emit electromagnetic radiation of any type, such asradio waves, infrared waves, etc. In other cases, the beacons may emitsound waves (e.g., in the ultrasound range). Further, in some cases, thebeacons may generate signals according any protocol or combination ofprotocols, such as the BLUETOOTH protocol, the Wi-Fi protocol, etc. Forexample, without limitation, the beacons may correspond to BLUETOOTH LowEnergy (BLE) beacons. Further, a beacon can have a range of any desireddepth, to best suit the target environment in which it is deployed. Insome indoor settings, for example, beacons are chosen having relativelyshort ranges, e.g., ranges of one or more meters. In some cases, abeacon's stated range may be based on the implicit or explicitly-statedassumption that its signals are to be detected by a particular type ofuser device (such as a particular smartphone), or a particular class ofuser devices (such as a particular class of smartphones), which haveknown and stable signal-receiving characteristics.

In yet another case, each beacon is a passive device (such as a passiveRFID) that may be interrogated by the user device 106 (or some otherinterrogating device) when that device is within a prescribed distanceto the beacon. That prescribed distance corresponds to the range of thedevice. However, to facilitate explanation, the remaining descriptionwill assume that each beacon actively emits a signal.

In one implementation, each beacon possesses a code which defines itidentity. The beacon may constantly or periodically (or in an on-demandmanner) emit signals that announce its code (and/or any otherapplication-specific information), which may be detected by receivingdevices within the range of the beacon. For example, consider a BLEbeacon that has a range of about one meter. A receiving device that lieswithin that range may detect the beacon's signal and read its particularcode.

In the example of FIG. 47, an indoor (and/or outdoor) environment ischaracterized by a collection of corridors and obstacles (e.g., obstacle1, obstacle 2, and obstacle 3). This scenario involves pre-populatingthe environment with a collection of beacons (e.g., beacons b₁, b₂, . .. b₇), e.g., by placing the beacons at every intersection of two or morecorridors. The outmost dashed-line circle surrounding each beaconrepresents the range of the beacon. As noted above, a receiving devicethat lies within the range of a beacon can successfully detect thebeacon; otherwise, the receiving device will have no knowledge of thebeacon. Note that, in one illustrative implementation, the ranges of thebeacons do not overlap. This characteristic is advantageous because iteliminates any ambiguity as to the location of the user at any giventime. In other words, the user's device cannot simultaneously detect twoor more beacons, because the ranges of these beacons do not overlap, andthe user cannot simultaneously exist at two places at the same time. Insome environments, the non-overlapping nature of the beacon ranges maybe ensured by also taking into consideration the nature of the userdevice (or class of devices) that will be receiving the signals emittedfrom the beacons.

As a first preliminary step, the locations of all beacons in theenvironment of FIG. 47 may be loaded in the data store 424. The datastore 424 may also store the codes associated with those beacons.

As a second preliminary step, the system 102 can use any route planningtool to generate a route through the environment of FIG. 47. The routeplanning tool can apply any algorithm to perform this task. For example,the routing planning tool can express the search space defined by thecorridors and obstacles as a graph having a plurality of nodes andlinks. That is, the links represent corridors and the nodes representintersections of corridors. The routing planning tool can then use anyalgorithm (such as the well-known Dijkstra's algorithm) to find the mostefficient path through the graph. Once the route is defined, thebeacon-based guidance module 422 can identify a subset of beacons(referred to herein as route-specific beacons) that will be traversed bythe user in traveling the route, and store the identities of thosebeacons. In the case of FIG. 47, the user is expected to encounter allof the beacons, but this is generally not the case (to be clarifiedbelow).

Now assume that the user embarks on the route using the above-describedsystem 102 as guidance. The system 102 may provide real-time guidance tothe user in the manner described above via the headset 108 and/or theuser device 106. In this case, however, the system 102 uses a differenttechnique to determine the user's current location at each time,compared to techniques described above.

More specifically, the beacon-based guidance module 422 constantly scansthe environment to determine whether the user is in the range of anybeacon. If so, the beacon-based guidance module 422 identifies thecurrent location of the user as the current location of the beacon thathas been detected. In other words, the beacon-based guidance module 422knows, a priori, the code associated with the detected beacon and itslocation in the environment. If the user's headset 108 or user device106 detects the presence of that beacon (based on the code conveyed bythe detected signal), then the beacon-based guidance module 422 may makethe assumption that the user has the same location as the detectedbeacon.

At this juncture, the beacon-based guidance module 422 can leverage theservices of the path guidance module 420 to direct the user in thedesired direction using a three-dimensional sound and/or based on otherguidance information (such as a non-three-dimensional sound, displayedinstructions, etc.). For example, assume that the user's user device 106detects that it is in the range of beacon b₄. The path guidance module420 will determine, based on the predetermined journey information, thatthe next waypoint along the user's journey corresponds to the beacon b₅.The path guidance module 420 can then generate a three-dimensional soundwhich the user will perceive as originating from the right side of theuser, which serves to direct the user towards the next waypoint (e.g.,beacon b₅). The user will interpret this cue as an instruction that heor she should turn to the right. The user will continue in thisdirection until he or she encounters another beacon (e.g., beacon b₅),at which time the direction of the user may be updated.

In some cases, a user may deviate in an unexpected manner from a plannedpath in such a manner that he or she falls outside the range of thebeacon that he or she was expected to encounter next. FIG. 48 representsone manner of addressing this situation. The strategy there is toincrease the number of beacons along the planned route of the user,which has the effect of increasing the frequency at which the user'scurrent position is assessed, and hence decreasing the potential thatthe user may wander off route (e.g., to the extent the he or she fallsoutside the range of a beacon that he or she is expected to encounter).Note that FIG. 48 shows only those beacons that the user is expected totraverse on the planned route. But the environment may includeadditional beacons (that are not shown) that the user is not expected totraverse.

The opportunity for a user to go significantly astray is not great inFIGS. 47 and 48, e.g., because the user's choices are significantlyrestricted by the obstacles. Nevertheless, a user may become confusedand take a wrong turn, causing him to leave the planned route. Or theuser may purposely decide to deviate from the planned path. For example,at the intersection associated with beacon b₉ in FIG. 48, the user maytake a left turn rather than a right turn. The user may thereforeeventually encounter an out-of-path beacon (not shown), which he is notexpected to encounter.

The beacon-based guidance module 422 can address this situation indifferent ways. In one case, the beacon-based guidance module 422 caninform the user that guidance can no longer be provided to the user,because the user appears to have wandered off track, and it is no longerpossible to determine the heading and intent of the user. Thebeacon-based guidance module 422 can also query the user whether he orshe intends to pursue the original defined path through the environment.

In another case, if sufficient information can be obtained regarding thecurrent location, heading, and intent of the user, the beacon-basedguidance module 422 can direct the user back onto the planned route. Forexample, the beacon-based guidance module 422 can determine thedirection that the user appears to be currently headed (although itappears to be wrong), by forming a trajectory based on a set of theuser's most recent known positions. The beacon-based guidance module 422can then use that heading information, together with informationregarding the user's current position (if known), to steer the user backonto the planned route. Or the beacon-based guidance module 422 canquery the user to determine whether he or she intends to still pursuethe planned route, or select another route.

In any scenario, the beacon-based guidance module 422 can also rely onother evidence of the current location and heading of the user (inaddition to information provided by the BLE beacons), when thatinformation is available. For example, the beacon-based guidance module422 can collect that information from GPS sensor sources, dead reckoningtechniques, etc. In other cases, it is assumed that at least some ofthese additional sources are not available, or are not reliable.

FIG. 49 describes another situation in which the space over which theuser traverses is more open-ended, compared to the examples of FIGS. 47and 48. The space is prepopulated with a collection of short-rangebeacons. For example, the space may be populated with a regular matrixof such beacons, if permitted by the physical characteristics of thespace. The beacons have ranges that do not overlap with each other.

Again, a route planning tool may generate a route 4902 through the spacebased on any input objective, and using any route planning algorithm forthis purpose. The beacon-based guidance module 422 then determines thebeacons that the user is expected to traverse as he or she travels theplanned route 4902. These beacons are represented as solid-black beaconsymbols.

During the actual traversal of the route 4902, the beacon-based guidancemodule 422 performs the same function as described above. That is, whenthe beacon-based guidance module 422 determines that the user hasentered the range of a beacon along the expected path (such as beaconb₂₂), then it accepts the location of that beacon as the currentlocation of the user. It then recalculates the heading of the user andupdates the three-dimensional beat sound (and/or other guidanceinformation) to guide the user towards the next beacon (e.g., beaconb₃₃).

The risk that the user will veer off track in the case of FIG. 49 isgreater than the case of FIG. 48 because the user is given more degreesof freedom in which the err. FIG. 50 describes one way of addressingthis situation. Here, the beacon-based guidance module 422 defines thebeacons (represented as solid-black beacon symbols) that the user mayencounter along a planned route 5002 in a more general manner, e.g., byalso encompassing neighboring beacons that lie to either side of themost optimal route 5002. The beacon-based guidance module 422 maycontinue to provide guidance to the user if he or she wanders into therange of these neighboring beacons, under the assumption that the useris still attempting to adhere to the planned route 5002, but has veeredonly slightly off course. The beacon-based guidance module 422 may onlygenerate an error condition when the user wanders beyond the boundariesassociated with outermost neighboring beacons.

In another implementation, the beacon-based guidance module 422 can forma postulate as to the desired destination of the user based on thebeacons that the user has already encountered along his or her path,thus far. For instance, the beacon-based guidance module 422 can form atrajectory based on the locations of the beacons encountered thus far.The beacon-based guidance module 422 can then determine a likelyintermediary or final destination to which the trajectory points, e.g.,by extending the trajectory along its current direction. Thebeacon-based guidance module 422 can then ask the user whether he or shewishes to pursue a path toward the assumed destination. If so, then thebeacon-based guidance module 422 can thereafter provide the same type ofnavigational assistance described above that helps the user reach theidentified destination.

In any of the cases described herein, the beacon-based guidance module422 can also take into consideration historical information regardingthe user's previous travel habits, and/or historical informationregarding the travel habits of others (with respect to a specifiedenvironment). That information, when available, may provide furtherevidence of the user's intent in reaching a desired destination.

FIG. 51 shows a process 5102 that describes one manner of operation ofthe beacon-based guidance module 422, e.g., within the context of thetypes of environments of FIGS. 47-50. In block 5104, the beacon-basedguidance module 422 receives a particular beacon signal from a sensor(or sensors) of a computing device which operates at a current locationwithin an environment. The computing device may correspond to the userdevice 108, or the headset 108, etc. As described above, the environmentis populated with a plurality of beacons having, in one illustrativeimplementation, respective non-overlapping ranges. Further, as apreliminary step, a route planning module may have defined a desiredroute, which is described by journey information. Or the desired routecan be generated in a dynamic manner as the user traverses theenvironment, based on assumptions that are made as to the intendedintermediary or final destination of the user. In any case, that desiredroute traverses ranges associated with a route-specific set of beacons,from among the total set of beacons in the environment.

In block 5106, the beacon-based guidance module 422 determines, based onthe particular beacon signal, whether the user is within a range of oneof the route-specific beacons; this operation yields current locationinformation when the user is within the range. In block 5108, thebeacon-based guidance module 422 determines a next waypoint that theuser is expected to reach, based on the predetermined journeyinformation, to provide next waypoint information. In some cases, thatnext waypoint may correspond to the next beacon along the user'spredetermined journey. In block 5110, the beacon-based guidance module422 next determines direction information based on the current locationinformation and the next waypoint information. The direction informationreflects a direction that the user is advised to travel to reach thenext waypoint. In block 5112, the beacon-based guidance module 422generates audio information (and/or other guidance information) based onthe direction information. In block 5114 the beacon-based guidancemodule 422 delivers the audio information to the user, e.g., as athree-dimensional beat sound. The audio information assists the user inreaching the next waypoint. The beacon-based guidance module 422 canperform the above functions with the assistance of the path guidancemodule 420 and the sound generation module 414.

FIG. 52 shows a process 5202 that provides further detail regarding onemanner in which the beacon-based guidance module 422 can determine thecurrent location of the user within an environment. In block 5204, thebeacon-based guidance module 422 identifies a particular beacon codeassociated with a particular beacon signal that has been received. Inblock 5206, the beacon-based guidance module 422 identifies, based onthe particular beacon code, a particular beacon that is associated withthe particular beacon code. In block 5208, the beacon-based guidancemodule 422 identifies a location of the particular beacon based onstored information (in the data store 424) which identifies the beaconcodes and respective locations of beacons within the environment.

In summary, the above features contribute to the above-stated goal ofallowing the user to safely and efficiently move through his or herenvironment, particularly in those situations in which the user cannotrely on other modes of determining his or her location (e.g., based onthe use of a satellite-based navigation system). In addition, the use ofnon-overlapping beacon ranges (according to one illustrativeimplementation) provides an efficient mechanism for disambiguating thelocation of the user, since the user cannot simultaneously exist withinthe ranges of two or more beacons at the same time.

In the above description of a first implementation, the assumption wasmade that the user device 106 (and/or the handset 108) receives, at anygiven time, a signal transmitted by either zero beacons or a singlebeacon, but not plural beacons. In other implementations, the abovecharacteristic can be relaxed in different ways.

For instance, in a second implementation, the beacon-based guidancemodule 422 will conclude the user device 106 and/or the handset 108 iswithin range of a particular beacon if it receives a signal from thatbeacon having a signal strength that is above a prescribed threshold.But unlike the first implementation, the beacon-based guidance module422 may also simultaneously receive weaker signals from one or moreother beacons in the environment, where the strength of each of thosesignals is below the prescribed threshold. In this scenario, theenvironment is populated with beacons having positions such that, at anygiven time, the beacon-based guidance module 422 will receive either:(1) no signal having a signal strength that is above the threshold; or(2) just one signal having a signal strength that is above thethreshold. In practice the second implementation functions in the samemanner as the first and offers the same benefits, e.g., by providing abinary mechanism for disambiguating the location of the user at anygiven time, assuming that the user is within the range of one of thebeacons. The beacons in the second implementation may therefore beconsidered as functionally or effectively non-overlapping due to theabove behavior. And accordingly, any reference to “non-overlapping” asused herein is to be understood as encompassing both the case in whichthe beacons have ranges that literally do not overlap, as well as thecase in which the ranges may be considered non-overlapping because theuser device 106 and/or headset 108 can at most receive a signal from onebeacon having a signal strength above the prescribed threshold.

In a third implementation, the beacon-based guidance module 422 mayreceive, at any given time, signals from any number of beacons havingany arbitrary signal strengths. The set of signals (and strengths) at aparticular location defines signal profile information for thatlocation. In a preliminary operation, the beacon-based guidance module422 can store signal profile information for each navigable location inthe environment, e.g., constituting information regarding the signalsand their respective strengths at that location. Collectively, thestored information constitutes a profile map of the environment. Duringnavigation, the beacon-based guidance module 422 can determine thesignals that it is receiving at a given time at a given location, toprovide current signal profile information. Then, the beacon-basedguidance module 422 can use the current signal profile information as akey to find the location having the closest matching signal profileinformation. That location defines the probable location of the user atthe given time. In some environments, the strengths of the signalsemitted by the beacons and/or the ability to detect those signals mayvary over time for various environment-specific reasons. Thebeacon-based guidance module 422 can address this issue by comparingnormalized versions of the signal profile information. Note that thepositions of the beacons in the third implementation need not meet thenon-overlapping constraints associated with the first or secondabove-described implementations.

F. Representative Computing Functionality

FIG. 53 shows computing functionality 5302 that can be used to implementany aspect of the system 102 set forth above. For instance, the type ofcomputing functionality 5302 shown in FIG. 53 can be used to implementany of the user device 106, any of the remote processing resources 112,the processing equipment used by the headset 108, the separate usercomputing device 110, and so on. In all cases, the computingfunctionality 5302 represents one or more physical and tangibleprocessing mechanisms.

The computing functionality 5302 can include one or more processingdevices 5304, such as one or more central processing units (CPUs),and/or one or more graphical processing units (GPUs), and so on.

The computing functionality 5302 can also include any storage resources5306 for storing any kind of information, such as code, settings, data,etc. Without limitation, for instance, the storage resources 5306 mayinclude any of RAM of any type(s), ROM of any type(s), flash devices,hard disks, optical disks, and so on. More generally, any storageresource can use any technology for storing information. Further, anystorage resource may provide volatile or non-volatile retention ofinformation. Further, any storage resource may represent a fixed orremovable component of the computing functionality 5302. The computingfunctionality 5302 may perform any of the functions described above whenthe processing devices 5304 carry out instructions stored in any storageresource or combination of storage resources.

As to terminology, any of the storage resources 5306, or any combinationof the storage resources 5306, may be regarded as a computer readablemedium. In many cases, a computer readable medium represents some formof physical and tangible entity. The term computer readable medium alsoencompasses propagated signals, e.g., transmitted or received viaphysical conduit and/or air or other wireless medium, etc. However, thespecific terms “computer readable storage medium” and “computer readablemedium device” expressly exclude propagated signals per se, whileincluding all other forms of computer readable media.

The computing functionality 5302 also includes one or more drivemechanisms 5308 for interacting with any storage resource, such as ahard disk drive mechanism, an optical disk drive mechanism, and so on.

The computing functionality 5302 also includes an input/output module5310 for receiving various inputs (via input devices 5312), and forproviding various outputs (via output devices 5314). Illustrative inputdevices include a keyboard device, a mouse input device, atouch-sensitive input device, a digitizing pad, one or more videocameras, one or more depth cameras, a free space gesture recognitionmechanism, one or more microphones, a voice recognition mechanism, anymovement detection mechanisms (e.g., accelerometers, gyroscopes, etc.),and so on. One particular output mechanism may include a presentationdevice 5316 and an associated graphical user interface (GUI) 5318. Otheroutput devices include a printer, a model-generating mechanism, atactile output mechanism, an archival mechanism (for storing outputinformation), and so on. The computing functionality 5302 can alsoinclude one or more network interfaces 5320 for exchanging data withother devices via one or more communication conduits 5322. One or morecommunication buses 5324 communicatively couple the above-describedcomponents together.

The communication conduit(s) 5322 can be implemented in any manner,e.g., by a local area network, a wide area network (e.g., the Internet),point-to-point connections, etc., or any combination thereof. Thecommunication conduit(s) 5322 can include any combination of hardwiredlinks, wireless links, routers, gateway functionality, name servers,etc., governed by any protocol or combination of protocols.

Alternatively, or in addition, any of the functions described in thepreceding sections can be performed, at least in part, by one or morehardware logic components. For example, without limitation, thecomputing functionality 5302 can be implemented using one or more of:Field-programmable Gate Arrays (FPGAs); Application-specific IntegratedCircuits (ASICs); Application-specific Standard Products (ASSPs);System-on-a-chip systems (SOCs); Complex Programmable Logic Devices(CPLDs), etc.

In closing, the following recitation summarizes respective aspects ofthe above-described system 102.

According to a first aspect, a system is described for assisting a userin interacting with a space. The system includes a headset forpresenting audio information to the user as the user interacts with thespace. The headset, in turn, has at least a set of input mechanisms forreceiving commands from the user, the command invoking respectivespace-interaction-related functions to be performed by a spaceinteraction module.

According to a second aspect, the headset uses a bone conductingmechanism to deliver the audio information to the user.

According to a third aspect, at least one command instructs the spaceinteraction module to start a listening mode. In the listening mode, thespace interaction module receives instructions that are verbalized bythe user.

According to a fourth aspect, at least one command instructs the spaceinteraction module to stop delivering the audio information.

According to a fifth aspect, at least one command instructs the spaceinteraction module to repeat one or more prior-delivered items of audioinformation.

According to a sixth aspect, at least one command instructs the spaceinteraction module to start an explore mode. In the explore mode, thespace interaction module provides information regarding a set of itemsof interest that are associated with a subspace to which attention ofthe user is currently directed.

According to a seventh aspect, at least one command instructs the spaceinteraction module to start an orientation mode. In the orientationmode, the space interaction module provides information regarding a setof items of interest that are associated with an entire space around theuser, at a current time.

According to an eighth aspect, at least one command instructs the spaceinteraction module to perform a more-information function. The spaceinteraction module performs the more-information function by providingadditional information regarding a topic associated with a previouslydelivered item of interest.

According to a ninth aspect, at least one input mechanism can invokeplural functions depending on a manner in which a user interacts withthe input mechanism.

According to a tenth aspect, the space interaction module is implementedby the headset.

According an eleventh aspect, the headset further includes one or moreof: a headset orientation determination mechanism for determining anorientation of the headset, to provide headset orientation information;a headset motion determination mechanism for determining a movement ofthe headset, to provide headset movement information; and/or a headsetposition determination mechanism for determining a position of theheadset, to provide headset position information.

According to a twelfth aspect, the system further comprises at least oneuser computing device. The user computing device provides a displayoutput mechanism for displaying information pertaining to interaction,by the user, with the space. The user computing device implements atleast part of functionality associated with the space interactionmodule. Further, the headset includes a first communication mechanismfor communicating with the user computing device, and the user computingdevice includes a second communication mechanism for communicating withthe headset.

According to a thirteenth aspect, the user computing device isconfigured to perform processing based on the headset orientationinformation and/or the headset movement information and/or the headsetposition information to generate a result, and to forward the result tothe headset.

According to a fourteenth aspect, the user computing device furtherincludes: a device orientation determination mechanism for determiningan orientation of the user computing device, to provide deviceorientation information; a device motion determination mechanism fordetermining a movement of the user computing device, to provide devicemovement information; and/or a device position determination mechanismfor determining a position of the user computing device, to providedevice position information.

According to a fifteenth aspect, the headset is configured to performfunctions based on the device orientation information and/or the devicemovement information and/or the device position information, as suppliedby the user computing device, when counterpart information is notavailable from the headset.

According to a sixteenth aspect, at least the headset and/or the usercomputing device is configured to receive information and/or to usefunctionality provided by an external computing device.

According to a seventeenth aspect, a method, implemented by a system, isdescribed for assisting a user in interacting with a space. The methodincludes: receiving an instruction based on actuation, by the user, ofat least one input mechanism that is coupled to a headset; performing afunction based on the instruction, to provide an output result; andbased on the output result, delivering audio information, via theheadset, which assists the user in interacting with the space.

According to an eighteenth aspect, the operation of performing thefunction entails performing the function using a space interactionmodule that is implemented, at least in part, by the headset.

According to a nineteenth aspect, the operation of performing thefunction entails performing the function using a space interactionmodule that is implemented, at least in part, by a separate usercomputing device which is communicatively coupled to the headset.

According to twentieth aspect, a headset is described that includes: anaudio output mechanism for presenting audio information that assists auser as the user interacts with a space; and a set of input mechanismsfor receiving commands from the user, the commands invoking respectivespace-interaction-related functions to be performed by a spaceinteraction module. At least one command instructs the space interactionmodule to start an explore mode, and at least one command instructs thespace interaction module to start an orientation mode. In the exploremode, the space interaction module provides information regarding a setof items of interest that are associated with a subspace to whichattention of the user is currently directed. In the orientation mode,the space interaction module provides information regarding a set ofitems of interest that are associated with an entire space around theuser, at a current time.

A twenty-first aspect corresponds to any combination (e.g., anypermutation or subset) of the above-referenced first through twentiethaspects.

According to a twenty-second aspect, one or more computing devices(and/or one or more headsets) are provided for implementing any of thefirst through twenty-first aspects.

According to a twenty-third aspect, a system is provided forimplementing any of the first through twenty-first aspects.

According to a twenty-fourth aspect, one or more computer readablestorage mediums are provided that include logic that is configured toimplement any of the first through twenty-first aspects.

According to a twenty-fifth aspect, one or more means are provided forimplementing any of the first through twenty-first aspects.

Further, in closing, the functionality described herein can employvarious mechanisms to ensure that any user data is handled in a mannerthat conforms to applicable laws, social norms, and the expectations andpreferences of individual users. For example, the functionality canallow a user to expressly opt in to (and then expressly opt out of) theprovisions of the functionality. The functionality can also providesuitable security mechanisms to ensure the privacy of the user data(such as data-sanitizing mechanisms, encryption mechanisms,password-protection mechanisms, etc.).

Further, although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. A system for assisting a user in interacting witha space, comprising: a headset for presenting audio information to theuser as the user interacts with the space, the headset having at least:a set of input mechanisms for receiving commands from the user, thecommand invoking respective space-interaction-related functions to beperformed by a space interaction module.
 2. The system of claim 1,wherein the headset uses a bone conducting mechanism to deliver theaudio information to the user.
 3. The system of claim 1, wherein atleast one command instructs the space interaction module to start alistening mode, wherein, in the listening mode, the space interactionmodule receives instructions that are verbalized by the user.
 4. Thesystem of claim 1, wherein at least one command instructs the spaceinteraction module to stop delivering the audio information.
 5. Thesystem of claim 1, wherein at least one command instructs the spaceinteraction module to repeat one or more prior-delivered items of audioinformation.
 6. The system of claim 1, wherein at least one commandinstructs the space interaction module to start an explore mode,wherein, in the explore mode, the space interaction module providesinformation regarding a set of items of interest that are associatedwith a subspace to which attention of the user is currently directed. 7.The system of claim 1, wherein at least one command instructs the spaceinteraction module to start an orientation mode, wherein, in theorientation mode, the space interaction module provides informationregarding a set of items of interest that are associated with an entirespace around the user, at a current time.
 8. The system of claim 1,wherein at least one command instructs the space interaction module toperform a more-information function, wherein the space interactionmodule performs the more-information function by providing additionalinformation regarding a topic associated with a previously delivereditems of interest.
 9. The system of claim 1, wherein at least one inputmechanism can invoke plural functions depending on a manner in which auser interacts with the input mechanism.
 10. The system of claim 1,wherein the space interaction module is implemented by the headset. 11.The system of claim 1, wherein the headset further includes one or moreof: a headset orientation determination mechanism for determining anorientation of the headset, to provide headset orientation information;a headset motion determination mechanism for determining a movement ofthe headset, to provide headset movement information; and/or a headsetposition determination mechanism for determining a position of theheadset, to provide headset position information.
 12. The system ofclaim 11, wherein the system further comprises at least one usercomputing device, the user computing device providing a display outputmechanism for displaying information pertaining to interaction, by theuser, with the space, wherein the user computing device implements atleast part of functionality associated with the space interactionmodule, and wherein the headset includes a first communication mechanismfor communicating with the user computing device, and the user computingdevice includes a second communication mechanism for communicating withthe headset.
 13. The system of claim 12, wherein the user computingdevice is configured to perform processing based on the headsetorientation information and/or the headset movement information and/orthe headset position information to generate a result, and to forwardthe result to the headset.
 14. The system of claim 12, wherein the usercomputing device further includes: a device orientation determinationmechanism for determining an orientation of the user computing device,to provide device orientation information; a device motion determinationmechanism for determining a movement of the user computing device, toprovide device movement information; and/or a device positiondetermination mechanism for determining a position of the user computingdevice, to provide device position information.
 15. The system of claim14, wherein the headset is configured to perform functions based on thedevice orientation information and/or the device movement informationand/or the device position information, as supplied by the usercomputing device, when counterpart information is not available from theheadset.
 16. The system of claim 12, wherein at least the headset and/orthe user computing device is configured to receive information and/or touse functionality provided by an external computing device.
 17. Amethod, implemented by a system, for assisting a user in interactingwith a space, comprising: receiving an instruction based on actuation,by the user, of at least one input mechanism that is coupled to aheadset; performing a function based on the instruction, to provide anoutput result; and based on the output result, delivering audioinformation, via the headset, which assists the user in interacting withthe space.
 18. The method of claim 17, wherein said performing of thefunction entails performing the function using a space interactionmodule that is implemented, at least in part, by the headset.
 19. Themethod of claim 17, wherein said performing of the function entailsperforming the function using a space interaction module that isimplemented, at least in part, by a separate user computing device whichis communicatively coupled to the headset.
 20. A headset, comprising: anaudio output mechanism for presenting audio information that assists auser as the user interacts with a space; a set of input mechanisms forreceiving commands from the user, the commands invoking respectivespace-interaction-related functions to be performed by a spaceinteraction module, wherein at least one command instructs the spaceinteraction module to start an explore mode, wherein, in the exploremode, the space interaction module provides information regarding a setof items of interest that are associated with a subspace to whichattention of the user is currently directed, wherein at least one othercommand instructs the space interaction module to start an orientationmode, and wherein, in the orientation mode, the space interaction moduleprovides information regarding a set of items of interest that areassociated with an entire space around the user, at a current time.